GitNexus/docs
Gergő Magyar cc6225d2cb
feat(analyze): make indexing resource-bounded and resumable (#3829)
* docs(plans): add resumable analyze plan

* perf(bench): measure resumable storage primitives

* feat(embeddings): bound and persist work selection

* feat(embeddings): persist resumable vector jobs

Keep graph readers independent of vector writes. Acknowledge units only after complete chunk verification and a native checkpoint.

* feat(embeddings): stream resumable job units

* feat(storage): add generation publication

Publish durable graph generations independently of vector jobs and gate readers on matching identities. Retain the legacy graph inode under a private name so older binaries cannot serve a stale canonical graph. Coordinator integration follows separately.

Refs #3598

* feat(embeddings): coordinate durable jobs

Join published graph selection, durable work, bounded native handles and vector publication behind one inactive controller. Limit optional HTTP response decoding within the job reservation and admit supported large inputs under larger budgets.

Refs #3598

* feat(analyze): pass phase resource allocations

Carry per-run worker heaps and native pools without changing legacy defaults. Hold parser reservations through safe native exit; refuse database reallocation after an unverified close.

* feat(embeddings): reserve sidecar resources

Pin each durable job to an admitted model heap and CPU config. Retain native allocations after caller cancellation until response or process exit, and reject phase handoff when release is unverified.

* feat(status): separate graph and vector readiness

Report embedding debt independently from graph content freshness. Read bounded durable progress without loading native runtimes, gate stale vector capability stamps, and allowlist additive phase counters on public SSE and ops payloads.

* fix(embeddings): recover interrupted job setup

Recover an owned empty native catalog after interrupted bootstrap, and rebuild damaged failure accounting without repeating completed provider work. Preserve foreign artifacts and propagate filesystem errors.

* fix(resources): verify native pool release

Keep failed and concurrent native closes tracked until release succeeds. Refuse another reserved allocation while cleanup is unverified, including failed connection construction. Let graph readers tear down after result-close errors.

* fix(resources): drain graph before phase admission

Wait for session work and verify the prior main graph pool has closed before parser or embedding admission. Recheck cancellation after the drain so cancelled jobs do not read or mutate job metadata.

* docs(plans): revise resumable analyze seams

* perf(bench): measure paired analyze runs

* feat(index): publish logical graph generations

Retain private database files across incremental generations and fence prepared publication against superseding writers. Exit-only producers must exit before publication; interrupted in-place writes remain explicitly dirty.

* feat(index): guard graph process handoff

Wait for actual process exit and reacquire the ordinary slot lock before
publishing the prepared attempt. Recheck placement and the registered
name, and retain graph discoverability after cancellation.

Add a bounded nonterminal worker receipt without serializing the live
graph. Production phase activation remains separate.

* feat(cli): defer success across graph handoff

Finish live-graph skill generation before the child hands off. Transfer
only bounded counters, identity and the pre-run self-commit safety flags;
wait for IPC delivery before returning for native process exit.

Reject missing supervisors and mismatched prepared results before skill
writes. Finalization, self-commit and success remain deferred until the
production supervisor completes embedding; core activation follows the
pending-artifact and resource contracts.

Refs #3598

* feat(embeddings): rebase pending paid artifacts

Separate graph job identity from mutable artifact ownership. Keep paid
compatible groups across source edits using bounded revalidation and
selection pruning; retain one sealed fallback for the active space.

* feat(embeddings): preserve historical coverage

* feat(embeddings): reuse verified legacy vectors

Retain a directly referenced legacy source across pending-artifact
rebases. Stream checkpoint exclusions and build a native metadata index
without copying vectors or retaining corpus-sized JavaScript maps.

Require recorded provider identity and provable complete input groups.
Keep unknown provenance explicit, source artifacts immutable, compatible
relocations free of provider calls, and new inputs in normal batches.

The controller remains inactive until coordinated writer activation.

Refs #3598

* feat(analyze): account for supervised phases

Charge parent memory once per phase. Carry allowlisted progress through
worker IPC and preserve counters across logs without completing the job.

Refs #3598

* fix(storage): retain owned analysis generations

Keep private generation files in place through shared-store preparation
and publication. Retain prepared publications and pending embedding work
across the unlocked process handoff, including work in branch slots.

Refs #3598

* fix(embeddings): share pinned vector readers

Share immutable vector file identities so sibling hard links retain one
native reservation. Keep the handle alive until its final reader closes,
including failed initialization and replacement of another reader's path.

Refs #3598

* feat(storage): share completed vector artifacts

Pin immutable vectors independently in checkout slots and shared graph snapshots. Keep mutable jobs with their owning checkout; resume interrupted adoption without overwriting unverified files.

Refs #3598

* feat(analyze): retain caches across phase handoff

Keep prepared staging during lock reacquisition and share cache
publication between the graph writer and exit continuation. Bound the
filesystem handoff to chunk keys; never copy live cache contents.

Refs #3598

* fix(analyze): retain owned generation routing

Keep prepared and pending work under its original slot lock during shared-store adoption. Preserve private generation filenames during incremental writes without copying the graph.

Drain model and native resources only when graph work actually starts, including FTS repair. Leave the warm no-op path unchanged.

Validated 158 tests, source build, test typecheck, formatting and lint. Complete graph gates cover 29 symbols in two files and the expected coordinator flow. End-to-end embedding activation and performance acceptance remain pending.

* feat(analyze): wire resumable phase execution

Publish verified graphs before embedding and hand off resource ownership
across CLI, watch, auto-sync, and HTTP phases. Keep completed vectors
readable while durable embedding units resume in a separate generation.

Preserve legacy vector reuse, avoid graph rebuilds for dimension changes,
and cover interruption boundaries with fresh-process restart tests.

Refs #3598

* fix(analyze): close recovery and ownership gaps

Retain interrupted private work, clear stale query widths, and keep
native reservations until confirmed worker exit. Keep shared-member
admission gates outside garbage-collected directories for flat and
branch writers. Reject invalid
routes and overlaps, and distinguish validation from uncertain writes.

Make test imports work before build, isolate host-capacity assertions,
and give legacy vector fixtures the production native pool allocation.

Focused regressions, clean-build types, formatting and lint pass.
Graph analysis maps 51 symbols across 21 paths. Two import-only test
paths remain unmapped; direct review and integration tests cover them.
The parent full suite passed 32,084 tests with five failures: two fixed
legacy-vector fixture cases and three recovery cases awaiting isolation.
Final full-suite and performance acceptance remain pending on the draft.

Refs #3598

* perf(embeddings): batch reused vector writes

Avoid a native transaction per reused node while retaining the byte
budget and durable unit boundary. Cover mixed inputs, cancellation,
and replay after read or committed-write failures.

Native regression fails before the fix (65 commits for 64 nodes).
79 focused tests and all three former recovery timeouts pass.
Fresh graph gate: 18 symbols, 2 files, 36 affected flows; complete
mapping. Final whole-command performance acceptance remains pending.

Refs #3598

* fix(analyze): honor explicit worker overrides

Preserve the existing CPU oversubscription override while reserving memory for every requested worker. Automatic concurrency stays CPU-bounded.

Validate on one- and four-CPU hosts: 81 tests pass; the override regressions fail before the fix.

* fix(storage): validate opened artifact files

Validate descriptors before reading bounded receipts and manifests. Use nonblocking opens to reject FIFO substitutions and verify directory entries when O_NOFOLLOW is unavailable. Preserve the metadata cache fast path and guard generation paths explicitly.

Remove the redundant pending-node fallback. Add race, symlink, cleanup, and cache regressions to platform coverage.

* fix(storage): guard the graph path directly

Apply the containment check to the same path passed to lstat. The nested normalization guarded a different expression in CodeQL. path.relative already resolves its inputs, so remove the redundant normalization and retain relative-path behavior.

* fix(embeddings): preserve AST inputs on resume

Carry indexed source policy into bounded resumable jobs and preserve exact-input reuse across sibling edits. Include strategy and retention in job identity; admit source memory before reading.

* chore: run checks for the rebased stack

* refactor(embeddings): simplify reuse plumbing

Name reuse-validation options and share the owned-directory loop.
Preserve callback order, directory sync, and ownership checks.

* test: fix recovery and Windows CI fixtures

Distinguish legacy checkpoints without input provenance from verified source-aware chunks. Check the embed API at its shared-phase boundary and model pathname races without rename-over-open on Windows.

* fix: close resource admission and CI gaps

Keep live sidecars charged after a reap timeout. Let sync and HTTP jobs use automatic host budgets instead of overcommitting to default V8 limits. Explain the incompatible native zero-pool override, clean up failed HTTP fixtures, and run the native vector suites across platforms.

* fix: preserve analysis resource ownership

Pin admitted model settings and retain worker and supervisor reservations.
Recheck canonical directory identity before receipt reads or writes.
Canonicalize the native publication fixture for macOS and Windows.

* test: distinguish parameterized CI cases

* docs: remove implementation plan from PR

* ci: gate resumable analyze performance

* fix(analyze): close recovery and CI gaps

Retain ownership receipts and paid vectors through interrupted work
and cleanup. Restore automatic heap and parser-worker sizing, and
cover publication, cancellation and capacity failures.

Refs #3829, #3598

Note: pre-existing setup.test.ts and setup-skills.test.ts failures
with inherited CODEX_HOME; all 62 tests pass with it unset for the
test process.

* fix(embeddings): normalize graph parent paths

Bind the canonical job root to the same graph path on macOS temp aliases
and Windows short paths. Keep graph identity and owned-directory checks.
Canonicalize the retention fixture and show subprocess stderr on failure.

Validation: 65 focused tests; 26 tests through an aliased TMPDIR; build,
test typecheck, formatting and lint; graph change analysis (4 files).

* fix(embeddings): batch writes and guard recovery

Batch bounded vector and receipt writes to remove per-node native calls
without changing validation, transactions, or durable checkpoints.

Pin publication descriptors before entry validation. Preserve excluded
vector coverage, reject malformed status paths, retry orphan sidecars,
and complete shutdown after child spawn failures.

Validation: 159 focused tests and 89 native/restart tests; build, test
 typecheck, lint and formatting. Refreshed graph analysis maps production
changes; three test files remain unmapped.

* ci: profile fresh analyze overhead

Capture process and parser-worker CPU profiles in a separate CI-only run. Reject profiling with --check so diagnostic overhead never affects the acceptance gate.

* perf(analyze): trim continuation startup work

* fix(analyze): preserve recovery and storage bounds

Keep automatic writer pools page-scaled and restore frontend heap caps.
Adopt a generation recovered during handoff, reclaim sealed artifacts,
and retire compatible legacy sources after migration completes.

Report semantic degradation and omit provider causes from API logs.
Enforce no paid replay in the CI resume benchmark.

* perf(analyze): defer unused parser imports

* fix(workers): retain failed shutdown reservations

* perf: skip unused budget supervisor receipt

* fix: bound rejection history and watch budgets

* perf: batch provider vector writes within units

* fix: harden cleanup and semantic availability

* perf: avoid repeated analysis startup probes

Skip unused automatic-heap parent provenance and reuse one npm version
probe for runner selection and its warning. Keep child/watch identity
capture and manual/test heap overrides. Refs #3829.

* fix: normalize legacy export paths on Windows

Cover Unicode export directories and restore the heap parser comment. Refs #3829.

* test: restore watch resource mock contract

Provide the supervisor reservation used by watch refresh admission. Refs #3829.

* fix: retain watch budgets and bound lookahead

Keep explicit heap reservations across refreshes and limit brace scans to the existing three-line window. Refs #3829.

* fix: stop retries on shutdown and allow empty parse

* refactor: simplify embedding setup and test lists

* perf: batch vector completion hash updates

Keep hash finalization inside the existing transaction and row bounds. Run embedding input-isolation tests in the serial native project.

* fix: address PR #3829 recovery feedback

Preserve terminal worker results, reclaim orphan attempt metadata, retain artifacts with unrecognized space ownership, and correct the cancellation comment.

* perf: retain simple vector completion query

* perf: skip redundant alias identity probes

* perf: avoid Git probes for ordinary checkout roots

* perf: avoid redundant vector transactions

Standalone writes already execute in automatic transactions. Keep multi-query transactions and durable unit checkpoints unchanged.

* docs: clarify cancellation grace period

* fix: finish vector recovery and explicit drops

Preserve paid peers through source-admission aborts and complete legacy retirement after interrupted publication. Explicit drops remove validated vector payloads before their ownership receipts so later runs regenerate them.

Refs #3829

* fix: preserve storage errors during aborts

Only unwrap aborts raised by source admission. A simultaneous native write failure must propagate instead of entering the provider-abort completion path.

Refs #3829

* fix: validate retained analysis artifacts

Reject malformed cache keys before shard writes and retain legacy work
before shared-store adoption. Explicit drops validate legacy ownership
and keep receipts until payload cleanup succeeds. Damaged registered
graphs block stale local hook fallback.

* fix: preserve legacy and preflight contracts

Keep the native-default pool override, retain incomplete legacy seeds
with their owner, and include resource preflight in the worker timeout.

* fix: honor resource and preflight boundaries

Release optional cache space before rejecting an admissible source read.
Start server deadlines before preflight and reject null progress counts.
Pass the native-default pool sentinel through the database opener.

* test: align native and drop contracts

Accept the documented native-default zero sentinel. Verify the rebuilt graph and empty vectors instead of an obsolete escalation log.

Refs #3829

* perf: avoid redundant vector checkpoints

Acknowledge verified WAL commits without rewriting the database per unit.
Keep the final publication checkpoint and prove crash recovery from WAL.
Give publication test cases distinct titles for the CI execution audit.

Refs #3829

---------

Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com>
2026-11-07 11:27:47 +00:00
..
languages docs: simplify guides and verify receiver gaps (#3694) 2026-10-23 09:06:03 +01:00
optimize fix(taint): avoid false positives for guarded Git arguments (#3703) 2026-10-24 15:37:26 +01:00
plans feat: add the test scenario map and a Vitest 5 mutation runner (#3686) 2026-10-27 16:46:58 +00:00
cli-reference.md feat(analyze): make indexing resource-bounded and resumable (#3829) 2026-11-07 11:27:47 +00:00
deployment.md feat(web): resolve a root-relative GITNEXUS_BACKEND_URL against the page origin (#3640) 2026-10-24 08:45:24 +01:00
editor-setup.md feat(opencode): ship hooks as auto-loaded global plugin for full support (#3619) 2026-10-28 13:45:08 +00:00
mcp-reference.md feat(taint): add AdonisJS request and Lucid models (#3727) 2026-11-02 11:04:56 +00:00
wiki-generation.md fix(wiki): configure limits and retry truncated pages (#3775) 2026-10-31 13:24:37 +00:00