GitNexus/gitnexus/bench/analyze-phase-breakdown.md

Analyze phase breakdown — where the time actually goes

Measured 2026-09-06/07 while landing #3194, #3196 and #3200. Records what the analyze pipeline costs per phase, and — as importantly — the optimizations that were measured and rejected, so the next person does not re-derive them.

Unlike parse-throughput.md (a synthetic-fixture scaffold), these numbers come from a real repo corpus. They are not a CI gate; the gate for the parse dispatch path is bench/parse-dispatch-rounds/.

---

Method, and one trap that invalidates everything

The corpus must be a git repository. A non-git checkout cannot record a schema fingerprint, so the tool forces a full rebuild on every run and prints:

non-git repositories never record a schema fingerprint, so this run rebuilds regardless

A "warm" run measured that way is a forced cold rebuild wearing a warm label. A 37.7s figure was recorded that way during this work and was meaningless. On a real git repo an unchanged re-analyze short-circuits to Already up to date.

And the analyzer binary must not change between runs. Rebuilding or re-copying dist changes the runner identity, and the tool then prints:

analyzer runner identity changed ...; forcing a full rebuild so the index provenance matches the analyzer

Every run after a rebuild is a full rebuild. To measure the incremental path, run once to stamp the identity, then edit and run again without touching dist. An earlier revision of this document reported full-rebuild numbers as if they were incremental for exactly this reason; they are corrected below.

And a mark on an await is not a mark on that call. The post-pipeline tail was timed by injecting timestamp marks into a disposable dist. An await that directly follows native work also absorbs whatever libuv still had queued, so the cost lands on the wrong line — that is how 845ms ended up attributed to a dynamic import that actually costs 0.035ms. Sanity-check any mark that lands on a call with no plausible work in it.

That is three separate "silently fall back to full work" guards — non-git corpus, runner identity, and the escalation gate. Read the banner on every run before trusting a number.

Corpus: this repository, git archive of HEAD into a scratch dir, then git init && git add -A && git commit. 5350 paths, 2234 parseable, ~30MB. 16 workers, dist on a local overlay filesystem (see "Filesystem" below). Phase numbers come from the ✓ Phase: <name> (<ms>) lines under NODE_ENV=development; the post-pipeline tail has no such lines and was measured by injecting timestamp marks into a disposable copy of the built dist.

---

Cold analyze

	before #3194	after #3196
total	110.3s	63.6s
parse	74.0s	36.0s
scopeResolution	17.0s	16.0s
all other phases	1.2s	1.2s
dispatches	221	15

Graph output identical throughout: 51,286 nodes / 163,092 edges / 2106 clusters / 759 flows.

Re-analyze after a one-file edit — the developer loop

Leaf file (cli/update-notice.ts, 2 importers), stable runner identity, so the incremental path is genuinely taken. 31.7s total.

step	ms	share
scopeResolution	~13900	44%
FTS index rebuild (buildSearchIndexesOrDegrade)	7525	24%
graph write (loadGraphToLbug, subgraph)	3566	11%
parse	~2800	9%
post-FTS event-loop drain (see below)	845	3%
everything else	~3000	9%

The 845ms was originally recorded against import('./platform/capabilities.js'). That import is not the cost: the CLI already imports the module statically, so a cached dynamic import measures 0.035ms and getRuntimeCapabilities() 0.1ms. The await there is the first yield after the native FTS build, so it absorbs whatever libuv work was still queued. Any mark placed on an await immediately after native work charges that work to the wrong line.

The incremental machinery works: the graph write was a 3,980-node subgraph, not the full 51,288. Everything the #3194/#3196 parse work optimized is ~9% of this.

A FULL-rebuild run of the same repo is 36-39s, with the graph write at ~6.3s and FTS at ~9.7s. Do not quote those as edit-loop numbers.

FTS index rebuild — the largest non-resolution cost, and it is a floor

A true incremental leaf edit rebuilds 8 of the 20 configured indexes — the tables the writeback actually DMLs. Per-index cost (measured against a copy of the corpus index, bench/ probe, reproduced within 2% of the in-analyze number):

 3541ms  File.file_fts        <- 47% of the incremental FTS cost on its own
 1696ms  Function.function_fts
  521ms  Method     506ms  Const      486ms  Property
  403ms  Interface  324ms  Class      291ms  TypeAlias
 ------
 7769ms  8 indexes   (in-analyze: 7525ms)

A FULL rebuild does all 20 and costs ~10.2s; the extra 12 indexes are only ~2.5s, so the narrowing is already doing its job. An earlier revision of this document called the narrowing "inconsistent" because one run rebuilt 8 and another 20 — the 20-index run was a forced full rebuild (the runner-identity trap above), not a leaf edit. There is nothing to fix there.

File.file_fts dominates because File rows carry whole-file content: 2442 rows, 32.9 MB, and Ladybug tokenizes at ~9.8 MB/s.

Four ways out, all measured, all closed

Narrow further — no. The 8 tables are exactly the ones holding rows for the 6 files in the write set (1 changed + 5 importers). There is no fat.

Build the indexes concurrently — impossible. A second connection issuing CREATE_FTS_INDEX fails immediately:

Cannot start a new write transaction in the system. Only one write transaction at a time is allowed in the system.

Eight builds on one connection serialize exactly (7886ms concurrent vs 7769ms serial).

Raise the connection's thread count — no effect. min-of-3 wall time at 4 / 8 / 16 / default(24) threads: 7298 / 7133 / 7109 / 7345 ms, inside the ~400ms per-config spread. CPU burned does move — 8.8s / 9.7s / 11.8s / 13.6s — so the default over-subscribes ~60% for nothing, but wall time is flat.

Skip File.file_fts when content did not change — cannot happen. Any file edit changes a File row, and Ladybug's FTS is not incremental: an index built before an insert does not see the new row, so a changed row forces a whole-table rebuild. Dropping content from the index takes it 3541ms → 241ms, but that is deleting full-file keyword search (#2317/#2323), not optimizing it. Capping the indexed content is a bad trade — the size distribution is flat, so a 64 KB cap still indexes 90% of the bytes while truncating the 72 largest files.

The one lever left: overlap

The FTS build runs on a libuv thread, not the main thread, and fully overlaps blocking JS:

index alone                    3859ms
index + 3000ms of JS burn      3337ms      (serial would be ~6859ms)

So the 3.5s File index could hide entirely behind the pipeline's ~17s of main-thread JS. File rows are the only ones that make this possible: they are { name, filePath } from processStructure, with content lazy-read from disk at CSV time, so they are fully determined by the file scan — before parsing, before resolution.

Two things block it today, and neither is small:

1. The DB is closed for the whole pipeline (closeLbug before runPipelineFromRepo, initLbug after). An early File write means holding a write handle across the pipeline.
2. It moves liveIndexMutationStarted before the pipeline. A pipeline failure would then leave the live index with fresh File content and stale symbols, instead of untouched.

scopeResolution — 14.7s, and it is two different problems

PROF_SCOPE_RESOLUTION=1 already exists and reports the internal split. Marks injected around the phase supply the rest. On a one-file edit:

step	ms	share
ParsedFile store rehydration	4426	30%
emit	7161	49%
resolve	966	7%
finalize	552	4%
extract	369	3%
per-language teardown, misc	~750	5%

extract is small because the parse cache works: 2121/2121 pre-extracted hits on the TypeScript pass. Nothing here re-parses.

Rehydration: three full store scans, one per language

The corpus resolves three languages — python (54 files), typescript (2121), javascript (59) — and loadParsedFilesForPaths walks every shard on each pass. The store is 413 shards / 301 MB. A pass that wants a single file costs 335 ms, because the skip decision needs the envelope's path listing and that listing is only trustworthy after the payload digest is checked.

So the fixed cost is paid per language, and it scales with language count, not with how many files that language has. Measured, min-of-5, three passes:

before   python 411ms   typescript 2872ms   javascript 507ms   = 3834ms
after    python 415ms   typescript 2805ms   javascript 248ms   = 3484ms

Memoizing each shard's authenticated listing for the run removes the repeat scans (−350 ms here; roughly −250 ms per additional language elsewhere). The first pass is unchanged by construction — it is what populates the memo.

What is left is a floor. The digest is not the cost: SHA-256 over all 301 MB takes 134 ms (2.25 GB/s, hardware-accelerated), so swapping it for CRC-32 would buy ~90 ms and cost an envelope-format bump. The remainder is v8.deserialize (~1240 ms) plus the cross-shard string intern walk (~1085 ms), and the intern walk is not optional — dropping it regresses retained heap ~59%, which is #2649's constraint.

emit is the real target

7.2s, 49% of the phase and ~21% of the whole edit loop. It is a fan of passes, each walking every reference site in the repo:

1828ms  emitCallableValueFlow      480ms  emitFreeCallFallback
1590ms  emitReceiverBoundCalls     426ms  emitReferencesViaLookup
 681ms  resolveDefGraphId          313ms  emitUniqueNamePropertyAccesses
 529ms  lookupCore                 280ms  emitReturnShapeMemberAccesses
 472ms  tryEmitEdge                222ms  emitPropertyDispatchCalls
 449ms  getScope

No hot inner loop, nothing quadratic, no single pass worth more than 12% of the phase. Micro-optimizing any of them is not the win.

The win is not running them. A one-file edit re-emits all 163,094 edges to write a 3,980-node subgraph. Every pass above runs over all 2121 TypeScript files because the pipeline rebuilds the full in-memory graph every run and only the DB write is incremental. Making emit incremental means knowing which files' edges can change when one file's registry contribution changes — which is a design, not a patch, and #2649 rules out "just cache the emitted edges".

---

Measured and rejected

Recorded because each cost real time to establish and each looks attractive from the armchair.

More workers buys nothing. Isolated-harness wall time at 16 / 20 / 24 workers: 44.1s / 44.8s / 43.3s — a 1.5s spread against a 3.7s within-size spread. A full-analyze sweep appeared to show 20 beating 16 by 6.3s; it was noise, and the sweep was invalid anyway because GITNEXUS_WORKER_POOL_SIZE was silently clamped at the time (fixed in #3200). DEFAULT_POOL_SIZE_CAP = 16 stands.

Bundling the worker entry buys ~250ms. Worker boot profiled at 12.2s per worker, of which getPackageScopeConfig 5.86s + internalModuleStat 3.14s + lstat/open ~2.2s — ESM module resolution, not native grammars (all 11 tree-sitter-* imports together are 33ms) and not V8 compile (53ms; NODE_COMPILE_CACHE gives zero gain). An esbuild bundle takes 16-worker boot 8.6s → 0.37s.

That was a filesystem artifact. On a normal overlay filesystem the same boot is 515ms stock vs 263ms bundled — 0.2% of a 110s analyze. The 8.6s only reproduces with the repo on a 9p mount (WSL2 D:\). Dropped.

Caveat carried by every number here: dist on a 9p mount costs ~3.5s of a 73s run (73.0s vs 69.6s on overlay). Measure on a local filesystem.

---

Open

The post-pipeline tail is fully accounted (99.7%): FTS rebuild, the graph write, and 0.8s of post-FTS event-loop drain between them.

FTS is closed as an optimization target except for the overlap above, which is a scheduling change to run-analyze.ts's open/close discipline rather than anything about FTS.

scopeResolution is now broken down. Its rehydration half has a floor and one repeat-scan win that is taken. Its other half — emit, 7.2s — is the largest remaining target in the whole edit loop, and the only way at it is incremental resolution.

Every optimization in this document that looked compelling from the armchair died under measurement. Measure first, and check the banner.