6 commits

Author	SHA1	Message	Date
Gergő Magyar	a665031c96	fix(lbug): diagnose staged-copy embedding failures, round max DB size (#3570) ... * fix(lbug): round GITNEXUS_LBUG_MAX_DB_SIZE up to a power of two The native buffer manager rejects any max DB size that is not a power of two, but the env parser accepted any positive integer, so a value such as 300000000000 failed at open and was then reported with a misleading "set a larger limit" hint. Round a positive safe integer up to the next power of two and warn once naming the raw and effective values. Values that are not safe integers now fall back to the 16 GiB default. Refs #3529 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * feat(lbug): add remedies for the address-space ceiling and past-EOF read Add two pure classifiers beside bufferPoolExhaustionRemedy. They parse only what the engine message states, label the byte count as the limit the engine reported (not the configured value), and name GITNEXUS_ATOMIC_INCREMENTAL=0 only for runs that knob changes. Neither asserts why the failure occurred. Refs #3529, #3530 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * feat(lbug): add size-evidence formatting for failure diagnostics formatFileEvidence is a pure formatter over stat-shaped input (apparent size, allocated blocks, raw statfs f_type), so sparse and regular cases are testable without a real sparse file. describeFileEvidence wraps it, never throws, and reports a missing file as unavailable. Refs #3530 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix(analyze): honor GITNEXUS_ATOMIC_INCREMENTAL=0 and classify staged failures GITNEXUS_ATOMIC_INCREMENTAL was only read as === '1', so the documented =0 mitigation did nothing for watch and auto-sync, which pass atomicIncremental as an option. An explicit =0 now beats the caller's request. Full rebuilds and the escalation to staging still stage and are unaffected. In the analyze catch block, snapshot the live and staging files before the checkpoint driver stops and the database closes (closing can checkpoint the staging file), classify the error once, then log the remedy and the held evidence for a size-ceiling or past-EOF failure. The off-switch is named only when the failed run was an atomic-incremental staged write. A max-size message the classifiers do not recognise keeps the generic hint, and the original error is rethrown unchanged. The atomic-incremental staging log line now records the staged file size. Refs #3529, #3530 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * docs: document max DB size rounding and GITNEXUS_ATOMIC_INCREMENTAL Note that GITNEXUS_LBUG_MAX_DB_SIZE must be a power of two and is rounded up, add GITNEXUS_ATOMIC_INCREMENTAL (1 and 0, and which builds 0 does not affect) to both README env tables, and add a Troubleshooting entry for the two intermittent embedding write-back failures and the evidence lines GitNexus now logs for them. Refs #3529, #3530 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix(review): apply review findings Review fixes: - Label the GITNEXUS_LBUG_BUFFER_POOL_SIZE=0 sentinel in the failure evidence line instead of printing "buffer pool 0 B". - Replace FailureRemedyContext {staged, offSwitchApplies} with a single StagedBuildMode ('in-place' | 'staged-rebuild' | 'staged-incremental'), removing the { staged: false, offSwitchApplies: true } illegal state. - Re-attach the LBUG_MAX_DB_SIZE doc comment to the exported constant. - Cover the in-place classified failure through runFullAnalysis, and the buffer-pool sentinel label, with new tests. Also carries the simplification pass: failure evidence is collected only for a classified failure, the staged-build flag is reused, the env var is read once, and the evidence tests share one temp-dir helper. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix(lbug): bound failure-evidence stats and record checkpoint activity Review findings #1 and #3 on #3570. describeFileEvidence raced nothing: a stat on a stalled 9p/NFS/virtiofs mount hung the analyze failure handler before the driver stopped, the database closed and the original error was rethrown. Each stat/statfs now has a 2000 ms deadline and fails open; a missing file reads "absent" instead of "unavailable", a stalled one reads "unavailable (timed out ...)". collectFailureEvidence (new) is the single evidence collector: it reads the checkpoint activity snapshot and dispatches every stat in one Promise.all before its first await, covering the live index, the staging copy and its .wal/.shadow/.wal.checkpoint sidecars. That is what lets a #3530 report tell a short main file from pages pending in the sidecars or from a CHECKPOINT GitNexus itself ran just before the failing DELETE. runCheckpointWithRetry now records completed count, last completion time and in-flight count. Both the periodic driver tick and checkpointOnce (the staged window-start CHECKPOINT) go through it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix(lbug): state the off-switch cost, pin the rounded max size, isolate ambient env Review findings #5 (note), #6 (wiring) and #11 on #3570. offSwitchNote now says that GITNEXUS_ATOMIC_INCREMENTAL=0 can leave the live index partly written after a failed in-place write, and that the next run then rebuilds it in full. A new test stubs GITNEXUS_LBUG_MAX_DB_SIZE=300000000000, reloads lbug-config and asserts the native Database constructor receives 2 ** 39 and that exactly one rounding warning is logged. The existing tests all called the parsers directly and matched the constructor argument with expect.any(Number), so reverting the wiring left the suite green. vitest.config.ts sets GITNEXUS_ATOMIC_INCREMENTAL to the empty string (neither '0' nor '1', i.e. unset) like GITNEXUS_MEMORY, so an exported =0 no longer switches off every staging test. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix(analyze): carry the failure diagnosis in the error message and label escalated runs Review findings #2, #4, #5, #7, #8 and the run-analyze half of #6 on #3570. #4: the remedy and `Failure evidence` lines were only sent to onLog, which `analyze --watch` wires only with GITNEXUS_VERBOSE=1 and the auto-sync launcher drops - the only two callers that request atomic-incremental staging. They are now appended to the rethrown error's message in place (identity and prototype preserved, native text first so classifiers anchored on it still match), which every caller prints. A thrown value that is not an Error falls back to a log. Auto-sync truncated that message to 240 characters before logging it; the log now allows 2000 while the persisted summary stays short. #8: remedy delivery is best effort, so a throwing onLog sink or an unwritable message can no longer replace the native error. #2: escalatedFullWrite is declared above the try, and an escalated atomic-incremental run is classified as a rebuild. Its remedy no longer points at GITNEXUS_ATOMIC_INCREMENTAL=0, which does not stop the escalation from staging. #5: when GITNEXUS_ATOMIC_INCREMENTAL=0 overrides a caller that requested staging, the run logs one line saying so and what a failed write costs. #7: the stage-time size line uses the same "staging copy" label as the failure evidence. #6: tests pin the evidence-before-driver-stop-before-close order, the exact configured max DB size in the evidence, and the appended text against the substring classifiers in the CLI and auto-sync. The READMEs describe the new evidence lines, ask reporters to include the stage-time line, and name GITNEXUS_WAL_MANUAL_CHECKPOINT=0 as the bisect knob for the checkpoint hypothesis. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Address PR review feedback (#3570) - wal-checkpoint-driver: count a checkpoint as completed only when tryFlushWAL actually ran one. It returns false when no connection is open, and the failure evidence renders the count as "N completed". - auto-sync runner: an oversized native error now logs its head and tail (1000 characters each) so the appended failure diagnosis, which is always the tail, survives. The persisted 240-character summary is unchanged. - lbug-config: the max-size remedy said GITNEXUS_LBUG_MAX_DB_SIZE "must be a power of 2", but non-powers are rounded up. It now says so, as do the matching lines in gitnexus/README.md. - failure-evidence and file-evidence tests use the default registered temp-dir prefix, so the stale-sidecar sweep recognises their fixtures. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Address PR review feedback (#3570) - run-analyze: stringify the thrown value through a helper that cannot throw, so an exotic throw (null-prototype object, throwing toString) cannot abort the failure handler before it stops the driver, closes the database and reclaims the staging copy. - run-analyze: a classified non-Error throw is wrapped in an Error (original kept as `cause`) so the diagnosis travels in the message like every other case, instead of in a log that the auto-sync launcher drops. This removes the separate log fallback. - auto-sync runner: the oversized-error log keeps head and tail within the 2000-character cap; the separator no longer pushes it to 2003. - lbug-config: the max-size rounding warning prints the operator's value as supplied (300000000000.9), not the integer it floored to. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Address PR review feedback (#3570) - lbug-config: the max-size remedy's rounding example was itself a power of two. It now shows a value being rounded: 300000000000 becomes 549755813888. - auto-sync-runner test: the comment said the whole log line is capped at 2000 characters; the assertion covers the error payload after the prefix. - extension-loader test: pin that the installer child is handed the rounded max DB size for a non-power-of-two override. The installer prefers that argument over the env var and has its own non-rounding parser, so this is what keeps a valid override working there. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Address PR review feedback (#3570) - Keep the whole failure diagnosis in auto-sync's capped log line: the tail now reserves the remedy and evidence lines (with a 240-char floor for the native head) instead of an even head/tail split. - Count only this run's checkpoints in the failure evidence: snapshot the process-global activity when the run begins and report the difference, so an earlier `analyze --watch` refresh no longer leaks in. - Diagnose classified failures thrown while opening the staging copy, via helpers shared with the write-path catch. - Halt checkpoint ticks before measuring failure evidence: start `walCheckpointDriver.stop()` (which clears its timer synchronously) before the stats, then await the drain before close. Note: pre-existing timeouts in run-analyze-fts-repair ("fails repair mode when no base meta exists", "abandons a pending set that fails EMBEDDING_RESUME_MAX_ATTEMPTS times in a row") reproduce on the unchanged PR head on this host and are not addressed by this commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>	2026-10-15 10:43:26 +01:00
Gergő Magyar	c474a811ba	fix(test): align fixtures and mocks with current types (#3471)	2026-10-05 04:47:37 +01:00
Gergő Magyar	e137daf63b	fix(auto-sync): allow self-hosted remotes via allowed_hosts (#3391) ... * fix(auto-sync): allow self-hosted remotes via allowed_hosts Auto-sync skipped any remote whose host was not github.com, gitlab.com, or gitee.com. Operators can now name exact extra DNS hosts in watch_config.yml without opening the default set. Co-authored-by: Cursor <cursoragent@cursor.com> * Address PR review feedback (#3391) - Dial auto-sync DNS names as absolute hosts and store that URL so a later fetch cannot follow a search domain. - Reject ambiguous numeric host spellings; an exact dotted IPv4 the operator listed stays opt-in. - Document allowed_hosts on the root auto-sync contract. Note: pre-existing failure in unit tests that require dist/cli/index.js and parse-worker.js (this worktree has no build); not addressed by this PR. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>	2026-09-27 10:18:57 +01:00
Gergő Magyar	6bb99767ff	fix(auto-sync): HTTPS remotes, OpenSSH image, and rc embeddings (#3378) ... * fix(docker): install OpenSSH in the CLI runtime image Auto-sync requires git SSH remotes, but the published image omitted openssh-client so every clone failed with ssh: not found. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(auto-sync): accept HTTPS remotes and reclone failed checkouts Allowlisted HTTPS URLs can clone without SSH keys, and a timed-out clone with no remote.origin is quarantined instead of blocking forever. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(auto-sync): honor .gitnexusrc embeddings and warn on empty vectors Auto-sync analyze now reads embeddings from the clone's project config, and query reports when an index has no vectors so keyword fallback is visible. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(review): warn when CodeEmbedding table is missing (U5) Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): wrap long openssh-client test line for prettier Co-authored-by: Cursor <cursoragent@cursor.com> * fix(query): keep keyword-only indexes off query.warning Empty or missing CodeEmbedding is the default index. Put the #3372 notice in a once-per-backend log line so FTS-success query results stay warning-free. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(review): bound rc reads and quarantine only a missing origin Drop the implementation plan from the branch. Auto-sync reads .gitnexusrc through the bounded control-file reader, and a git config failure no longer relocates a live checkout. The same allowlisted repo can switch between SSH and HTTPS without a refused pull. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(auto-sync): share repo identity and skip a second origin read Co-authored-by: Cursor <cursoragent@cursor.com> * test(auto-sync): clean temp fixtures and cover nested embeddings precedence Co-authored-by: Cursor <cursoragent@cursor.com> * test(auto-sync): skip the symlink rc fixture on Windows Co-authored-by: Cursor <cursoragent@cursor.com> * test(auto-sync): reject symlink rc files on every platform Co-authored-by: Cursor <cursoragent@cursor.com> * test(ci): run auto-sync symlink and clone tests on Windows and macOS Co-authored-by: Cursor <cursoragent@cursor.com> * test(git-clone): keep Windows CI on file URLs and POSIX permission checks Co-authored-by: Cursor <cursoragent@cursor.com> * test(git-clone): keep the SSH-to-HTTPS origin check offline Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>	2026-09-25 19:42:26 +01:00
Matt	8b21ce3b95	feat(auto-sync): preserve PDG indexes across updates (#3290) ... * feat(auto-sync): preserve PDG indexes across updates * docs(auto-sync): document durable PDG synchronization * Address PR review feedback (#3290) - Correct requestedPdg state docs for threshold-skipped syncs - Defer coalesced follow-up and skip failure-threshold counts for leftover-worker / retryable lock waits - Document the pdg tri-state and caveat the 30m/5m example Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): raise Windows Ladybug #605 hang budget off the CI tail Windows 3/3 typically finishes this native race in ~15s but has a 56s tail; 60s false-positives as deadlock. Keep the POSIX 60s detector and the completion/.shadow/row-count contract. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Gergő Magyar <gergomagyar@icloud.com> Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>	2026-09-16 20:12:45 +01:00
Yayler	3aa62be717	feat: add gitnexus auto-sync for scheduled remote clone and analyze (#2493) ... * adds an opt-in auto sync and analysis loop for GitNexus * adds an opt-in auto sync and analysis loop for GitNexus,gitnexus watch [init|start|restart|stop|status] * adds an opt-in auto sync and analysis loop for GitNexus,gitnexus watch [init|start|restart|stop|status] * fix: address PR review cleanup * Prettier code style * merge main * fix(watch): protect local repos and cancel active analysis * fix(watch): harden auto-sync lifecycle and locking - validate watch process identity before lifecycle operations\n- serialize registry, analysis, and LadybugDB access with recoverable locks\n- harden clone paths, symlinks, hooks, quarantine, and worker timeouts\n- install procps in the CLI image for reliable Docker watch control\n- add focused regression coverage for lifecycle, locks, clone, and registry behavior * update agents & claude md * merge main * fix(watch): harden auto-sync lifecycle * fix(watch): normalize SSH repo identity paths * fix(watch): normalize SSH repo identity paths * fix(watch): safely cancel analysis across platforms * fix(auto-sync): close worker and group sync failure paths * fix(auto-sync): drop retired allowStale from group sync allowStale was removed from SyncOptions, which broke typecheck and CI on this PR. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(watch): satisfy prefer-const and Prettier in auto-sync The watch timers are assigned exactly once, so prefer-const rejected the deferred `let` declarations. They are only read from `stop()` and the control poll, both of which run after the assignments, so binding them at creation is safe and drops the now-dead undefined guards. Remaining files are formatting only. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(watch): make lock identity absolute and stop three fail-open paths Lock owner identity was rendered by `ps -o lstart=` through localtime and the active locale, so the same live process produced a different string under a different TZ. A mismatch reads as PID reuse, so one daemon could reclaim a mutex another still held. Pin TZ=UTC and LC_ALL=C. The owner record also carried no hostname, so a holder on another machine was judged by this kernel's view of its PID — always "stale" — and its lock stolen whenever GITNEXUS_HOME is a shared volume. Record and compare the hostname, as the index lock already does. Ownership verification threw unconditionally on win32, which is reached once per project per tick, so watch reported `running` and then failed every repo forever. POSIX uid/mode cannot be checked there; skip those two assertions and keep the dangerous-root, symlink, containment and internal-root guards. Also: quarantine sweep now refuses a symlinked root instead of deleting through it; an unreadable state file propagates instead of being rewritten as empty state, which used to erase every repo's analyzed commit and failure count; a failed staging cleanup no longer strands a published lock with no release handle; and the concurrency runner settles every worker before surfacing a failure so cancellation cannot orphan a live analyze fork. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(watch): land the deferred review findings Six findings that were deferred from the review backlog, plus the docs they change. Worker heap: admission allowed `floor(availableMemoryGB / 2)` slots while every fork was handed the whole machine's heap cap, so the budget meant nothing as soon as an operator raised max_concurrency. Divide the cap by the repos actually analyzed in parallel. The default single-project path is unchanged. Registration: the parent registered without a branch, so it always took the primary/flat arm and relabelled a pinned branch entry on the branch-fallback path. Reproduce the worker's own resolveBranchPlacement decision instead. Cancellation: requestCancellation cleared the only timer and settled nothing, so a worker wedged past its safe point left the promise pending forever, wedging activeRun and hanging `watch stop`. Add a 5s grace after which the parent stops waiting and releases the IPC channel's hold on its event loop. The child is still never killed — it may be inside native work. overwrite_local_changes: `checkout --force` rewrites tracked files only, so untracked sources survived and were indexed as if they came from the remote. `git clean -fd -e /.gitnexus` after checkout; no -x/-X, so ignored paths and GitNexus's own storage survive. Quarantine: age alone never bounds a repo that fails every tick, since each partial clone is younger than the retention window. Keep the five newest per repo. Validation: repo_git_timeout is now bounded by the lesser of an hour and the sync interval, which is also the guard for the bare-number-means-seconds slip (`600000` meant ~7 days and cleared the timer ceiling). And the remote URL's final segment is validated at config load rather than failing once per tick inside the sync loop. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(watch): release an errored worker, and stop rejecting dotted repo names Three findings from the latest review pass. The 'error' handler settles immediately rather than waiting out the grace, so cleanup() clears the grace timer that would otherwise have released the child. An errored IPC channel does not mean the worker stopped, so release it on that path too — still no kill. The traversal guard tested the raw path for '..', which also rejected an ordinary name like owner/foo..bar that the repository-name rule accepts. Traversal is a whole segment, so test segments. The heap-cap test left two runs and their real timers pending; it now stubs timers and settles both promises. Registration coverage now pins the branch slot rather than leaving it implicit. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(watch): validate namespace segments and pin the stopped process identity Replacing the raw-string `..` test with a per-segment one dropped a guard: a segment like `..\..\outside` is not literally `..`, so it passed, and those segments build the clone path — on Windows the backslashes are separators. Hold every namespace segment to the same charset as the repo name, which keeps a separator out of a segment while still allowing an ordinary `foo..bar`. The final segment keeps its own check so a bad repo name keeps its own message. The stop wait polled liveness by pid alone, so a pid reused mid-wait would have it wait on an unrelated process and then report the watch stopped. Compare the process start time recorded for the owner, which also returns sooner. Registration now omits `branch` for a primary index instead of passing it as undefined, so that call keeps the shape it had before this branch. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cli): ship auto-sync as the remote daemon, reserve gitnexus watch. Keep analyze --watch for local incremental re-index and stop the top-level watch verb from starting a clone/pull loop. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(auto-sync): reject invalid branch refs and verify status identity (#2493) Reject leading slashes and per-component trailing dots in configured branches, and verify the live watch owner before trusting a stored error status. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(auto-sync): reject ownerIds that can escape the watch directory (#2493) Stop interpolating a tampered ownerId into the stop-request filename; only basename-safe values are treated as owners. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(auto-sync): recognize auto-sync in the watch-process identity check (#2493) Stop/status were still looking for a standalone watch token after the command rename, so a live gitnexus auto-sync start process would be refused as unrelated. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(autofix): apply prettier + eslint fixes via /autofix command * fix(auto-sync): reject boolean max_concurrency instead of coercing it to 1 (#2493) Number(true) is 1, so a YAML boolean would have passed the integer check and silently meant one worker. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(auto-sync): swallow status errors in the watch finally path (#2493) An uncaught updateStatus rejection in finally became an unhandled rejection. Skip the clone-root symlink test on Windows, where directory symlinks need privileges. Align the group-lock comment with fail-closed registry timeouts. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(auto-sync): catch cancelling status-write failures (#2493) Fire-and-forget updateStatus('cancelling') could become an unhandled rejection, the same class as the finally-path status write. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(auto-sync): ignore queued interval ticks after stop (#2493) clearInterval does not cancel a timer callback already queued. Guard runSafely on stopping so shutdown cannot start a new un-cancellable run. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(auto-sync): report stored watch status timestamps (#2493) status should show when the watch last entered a state, not when the CLI queried it. The failure-count test still expects 1 after a new commit resets the streak; rename it so that reset is explicit. Co-authored-by: Cursor <cursoragent@cursor.com> * style(auto-sync): apply prettier to starter status logger (#2493) Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: weiyf <weiyf3634@163.com> Co-authored-by: Gergő Magyar <gergomagyar@icloud.com> Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-09-02 19:40:18 +01:00