mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-11-11 08:37:35 +00:00
3 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Gergő Magyar
|
a665031c96
|
fix(lbug): diagnose staged-copy embedding failures, round max DB size (#3570)
* fix(lbug): round GITNEXUS_LBUG_MAX_DB_SIZE up to a power of two The native buffer manager rejects any max DB size that is not a power of two, but the env parser accepted any positive integer, so a value such as 300000000000 failed at open and was then reported with a misleading "set a larger limit" hint. Round a positive safe integer up to the next power of two and warn once naming the raw and effective values. Values that are not safe integers now fall back to the 16 GiB default. Refs #3529 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * feat(lbug): add remedies for the address-space ceiling and past-EOF read Add two pure classifiers beside bufferPoolExhaustionRemedy. They parse only what the engine message states, label the byte count as the limit the engine reported (not the configured value), and name GITNEXUS_ATOMIC_INCREMENTAL=0 only for runs that knob changes. Neither asserts why the failure occurred. Refs #3529, #3530 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * feat(lbug): add size-evidence formatting for failure diagnostics formatFileEvidence is a pure formatter over stat-shaped input (apparent size, allocated blocks, raw statfs f_type), so sparse and regular cases are testable without a real sparse file. describeFileEvidence wraps it, never throws, and reports a missing file as unavailable. Refs #3530 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix(analyze): honor GITNEXUS_ATOMIC_INCREMENTAL=0 and classify staged failures GITNEXUS_ATOMIC_INCREMENTAL was only read as === '1', so the documented =0 mitigation did nothing for watch and auto-sync, which pass atomicIncremental as an option. An explicit =0 now beats the caller's request. Full rebuilds and the escalation to staging still stage and are unaffected. In the analyze catch block, snapshot the live and staging files before the checkpoint driver stops and the database closes (closing can checkpoint the staging file), classify the error once, then log the remedy and the held evidence for a size-ceiling or past-EOF failure. The off-switch is named only when the failed run was an atomic-incremental staged write. A max-size message the classifiers do not recognise keeps the generic hint, and the original error is rethrown unchanged. The atomic-incremental staging log line now records the staged file size. Refs #3529, #3530 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * docs: document max DB size rounding and GITNEXUS_ATOMIC_INCREMENTAL Note that GITNEXUS_LBUG_MAX_DB_SIZE must be a power of two and is rounded up, add GITNEXUS_ATOMIC_INCREMENTAL (1 and 0, and which builds 0 does not affect) to both README env tables, and add a Troubleshooting entry for the two intermittent embedding write-back failures and the evidence lines GitNexus now logs for them. Refs #3529, #3530 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix(review): apply review findings Review fixes: - Label the GITNEXUS_LBUG_BUFFER_POOL_SIZE=0 sentinel in the failure evidence line instead of printing "buffer pool 0 B". - Replace FailureRemedyContext {staged, offSwitchApplies} with a single StagedBuildMode ('in-place' | 'staged-rebuild' | 'staged-incremental'), removing the { staged: false, offSwitchApplies: true } illegal state. - Re-attach the LBUG_MAX_DB_SIZE doc comment to the exported constant. - Cover the in-place classified failure through runFullAnalysis, and the buffer-pool sentinel label, with new tests. Also carries the simplification pass: failure evidence is collected only for a classified failure, the staged-build flag is reused, the env var is read once, and the evidence tests share one temp-dir helper. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix(lbug): bound failure-evidence stats and record checkpoint activity Review findings #1 and #3 on #3570. describeFileEvidence raced nothing: a stat on a stalled 9p/NFS/virtiofs mount hung the analyze failure handler before the driver stopped, the database closed and the original error was rethrown. Each stat/statfs now has a 2000 ms deadline and fails open; a missing file reads "absent" instead of "unavailable", a stalled one reads "unavailable (timed out ...)". collectFailureEvidence (new) is the single evidence collector: it reads the checkpoint activity snapshot and dispatches every stat in one Promise.all before its first await, covering the live index, the staging copy and its .wal/.shadow/.wal.checkpoint sidecars. That is what lets a #3530 report tell a short main file from pages pending in the sidecars or from a CHECKPOINT GitNexus itself ran just before the failing DELETE. runCheckpointWithRetry now records completed count, last completion time and in-flight count. Both the periodic driver tick and checkpointOnce (the staged window-start CHECKPOINT) go through it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix(lbug): state the off-switch cost, pin the rounded max size, isolate ambient env Review findings #5 (note), #6 (wiring) and #11 on #3570. offSwitchNote now says that GITNEXUS_ATOMIC_INCREMENTAL=0 can leave the live index partly written after a failed in-place write, and that the next run then rebuilds it in full. A new test stubs GITNEXUS_LBUG_MAX_DB_SIZE=300000000000, reloads lbug-config and asserts the native Database constructor receives 2 ** 39 and that exactly one rounding warning is logged. The existing tests all called the parsers directly and matched the constructor argument with expect.any(Number), so reverting the wiring left the suite green. vitest.config.ts sets GITNEXUS_ATOMIC_INCREMENTAL to the empty string (neither '0' nor '1', i.e. unset) like GITNEXUS_MEMORY, so an exported =0 no longer switches off every staging test. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix(analyze): carry the failure diagnosis in the error message and label escalated runs Review findings #2, #4, #5, #7, #8 and the run-analyze half of #6 on #3570. #4: the remedy and `Failure evidence` lines were only sent to onLog, which `analyze --watch` wires only with GITNEXUS_VERBOSE=1 and the auto-sync launcher drops - the only two callers that request atomic-incremental staging. They are now appended to the rethrown error's message in place (identity and prototype preserved, native text first so classifiers anchored on it still match), which every caller prints. A thrown value that is not an Error falls back to a log. Auto-sync truncated that message to 240 characters before logging it; the log now allows 2000 while the persisted summary stays short. #8: remedy delivery is best effort, so a throwing onLog sink or an unwritable message can no longer replace the native error. #2: escalatedFullWrite is declared above the try, and an escalated atomic-incremental run is classified as a rebuild. Its remedy no longer points at GITNEXUS_ATOMIC_INCREMENTAL=0, which does not stop the escalation from staging. #5: when GITNEXUS_ATOMIC_INCREMENTAL=0 overrides a caller that requested staging, the run logs one line saying so and what a failed write costs. #7: the stage-time size line uses the same "staging copy" label as the failure evidence. #6: tests pin the evidence-before-driver-stop-before-close order, the exact configured max DB size in the evidence, and the appended text against the substring classifiers in the CLI and auto-sync. The READMEs describe the new evidence lines, ask reporters to include the stage-time line, and name GITNEXUS_WAL_MANUAL_CHECKPOINT=0 as the bisect knob for the checkpoint hypothesis. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Address PR review feedback (#3570) - wal-checkpoint-driver: count a checkpoint as completed only when tryFlushWAL actually ran one. It returns false when no connection is open, and the failure evidence renders the count as "N completed". - auto-sync runner: an oversized native error now logs its head and tail (1000 characters each) so the appended failure diagnosis, which is always the tail, survives. The persisted 240-character summary is unchanged. - lbug-config: the max-size remedy said GITNEXUS_LBUG_MAX_DB_SIZE "must be a power of 2", but non-powers are rounded up. It now says so, as do the matching lines in gitnexus/README.md. - failure-evidence and file-evidence tests use the default registered temp-dir prefix, so the stale-sidecar sweep recognises their fixtures. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Address PR review feedback (#3570) - run-analyze: stringify the thrown value through a helper that cannot throw, so an exotic throw (null-prototype object, throwing toString) cannot abort the failure handler before it stops the driver, closes the database and reclaims the staging copy. - run-analyze: a classified non-Error throw is wrapped in an Error (original kept as `cause`) so the diagnosis travels in the message like every other case, instead of in a log that the auto-sync launcher drops. This removes the separate log fallback. - auto-sync runner: the oversized-error log keeps head and tail within the 2000-character cap; the separator no longer pushes it to 2003. - lbug-config: the max-size rounding warning prints the operator's value as supplied (300000000000.9), not the integer it floored to. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Address PR review feedback (#3570) - lbug-config: the max-size remedy's rounding example was itself a power of two. It now shows a value being rounded: 300000000000 becomes 549755813888. - auto-sync-runner test: the comment said the whole log line is capped at 2000 characters; the assertion covers the error payload after the prefix. - extension-loader test: pin that the installer child is handed the rounded max DB size for a non-power-of-two override. The installer prefers that argument over the env var and has its own non-rounding parser, so this is what keeps a valid override working there. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Address PR review feedback (#3570) - Keep the whole failure diagnosis in auto-sync's capped log line: the tail now reserves the remedy and evidence lines (with a 240-char floor for the native head) instead of an even head/tail split. - Count only this run's checkpoints in the failure evidence: snapshot the process-global activity when the run begins and report the difference, so an earlier `analyze --watch` refresh no longer leaks in. - Diagnose classified failures thrown while opening the staging copy, via helpers shared with the write-path catch. - Halt checkpoint ticks before measuring failure evidence: start `walCheckpointDriver.stop()` (which clears its timer synchronously) before the stats, then await the drain before close. Note: pre-existing timeouts in run-analyze-fts-repair ("fails repair mode when no base meta exists", "abandons a pending set that fails EMBEDDING_RESUME_MAX_ATTEMPTS times in a row") reproduce on the unchanged PR head on this host and are not addressed by this commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
Gergő Magyar
|
21a52af1d4
|
fix(lbug): ship FTS per-platform and recover in-place native aborts (#3274)
* fix(lbug): pin Ladybug core so Dependabot cannot ship a skewed FTS artifact The extension version is a separate upstream constant. Ignore daily core bumps and fail the pairing gate when the committed manifest does not name the installed core. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): make doctor and CI FTS gates resolve the packaged artifact Doctor and the REQUIRE_FTS file gates still treated an empty ~/.lbdb as unavailable, which would turn three CI jobs red once analyze stops installing into that tree. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): name native-abort and tuple-missing so analyze cannot mis-advise The CLI summary's trailing else treated every unknown skip reason as a missing extension. New crash and platform causes must get their own remedies, not a network-install hint. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): delete the dead read-path FTS index create ensureFTSIndex had no production callers and swallowed read-only CREATE_FTS_INDEX failures, which hid the only signal that a reader tried to write. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): vendor per-platform FTS artifacts so analyze needs no host install Keyword search depended on a CDN fetch into ~/.lbdb. Shipping the five published tuples inside the package makes air-gapped and ignore-scripts installs load the same artifact the publish gate checksums. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): load the packaged FTS artifact before any network install Analyze still required a CDN fetch into ~/.lbdb even when the package already shipped the file. FTS now path-loads the vendored tuple first and records source labels so a later truncated home copy cannot steal the diagnosis. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): diagnose a core/extension version skew instead of a missing runtime A structurally valid FTS artifact whose path version disagrees with the packaged pin must name both versions, not prescribe VC++ or OpenSSL. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): stamp an FTS phase so repair stays usable after an in-place abort A native CREATE_FTS_INDEX abort leaves no skip reason; the next run infers it from the dirty flag, and --repair-fts must not treat that phase as a half-written graph. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): park an in-place FTS crash WAL without wiping the graph An FTS abort after a successful checkpoint must reopen the live index on macOS, Windows, and Linux. Staging never parks the live WAL; readers keep today's large-WAL refusal. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): refuse read-only opens of an FTS-poisoned WAL MCP and serve cannot repair a leftover in-place abort. Fail before the native open and name --repair-fts, on macOS, Windows, and Linux. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): name a vendor-neutral Windows OpenSSL prerequisite OQ1 is unanswered here so GitNexus does not ship OpenSSL DLLs. Windows FTS now asks for a system OpenSSL 3 runtime instead of Git Bash PATH. Co-authored-by: Cursor <cursoragent@cursor.com> * test(lbug): inject the FTS vendor root and redact it on HTTP and MCP Path-loaded artifacts no longer vary with HOME. Tests pass an injected vendor tree and assert search warnings never leak a filesystem path. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(lbug): document load-only as the global FTS install default Analyze still overrides to auto. Packaged per-platform artifacts load before any network install on macOS, Windows, and Linux. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(lbug): format the FTS install-policy README table Prettier does not run on Markdown in pre-commit, so the U10 table wrap needs its own formatting commit. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): skip FTS CREATE after a persisted native abort A recovered analyze run was retrying CREATE_FTS_INDEX from skipReason alone. Keep that skip until --repair-fts, fail closed on unsupported tuples, and honor the checkpoint warrant for park/repair. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): honor checkpoint flushed warrant and align FTS tests with packaged vendor A no-op CHECKPOINT must not satisfy the FTS park warrant, and CI still asserted HOME-only FTS isolation after analyze started path-LOADing the packaged artifact. Co-authored-by: Cursor <cursoragent@cursor.com> * test(lbug): accept a nonempty incremental write set in the #2790 recovery check FTS-phase recovery can incremental-add files (changed=0, added=1). That is not the #2790 empty-diff wipe skip. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): compare FTS home versions to the core pin and tighten the publish filename gate Ladybug's ~/.lbdb/extension directory is the runtime/core version; treating it as the artifact version false-diagnosed skew. The publish guard now rejects a path-escaping filename the same way the fetch script does. Co-authored-by: Cursor <cursoragent@cursor.com> * test(lbug): seed FTS e2e fixtures from the packaged vendor artifact A machine with no ~/.lbdb copy should still run the vendor-survivorship cases; the seed no longer depends on HOME or a network install. Co-authored-by: Cursor <cursoragent@cursor.com> * Address PR review feedback (#3274) Keep in-place FTS abort evidence after persist so a second CREATE abort cannot fail-open readers, and close the CLI, loader, embed, and e2e gaps the review called out. Note: full npm test hit Ladybug worker-pool startup failures under memory pressure; tsc and 180 targeted unit tests passed. Co-authored-by: Cursor <cursoragent@cursor.com> * Address PR review feedback (#3274) Run the vendored-path symlink guard on the OS matrix, put e2e HOME fixtures on Ladybug's real extension layout, pin the embed crash-WAL gate before the writable open, and let analyze writers park through missing-shadow recovery. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lbug): keep --repair-fts CI green after vendored-first FTS Never-installed warning fixtures must not inspect a packaged vendor binary, and a failed dirty restamp must not abort an otherwise successful --repair-fts run. Co-authored-by: Cursor <cursoragent@cursor.com> * test(cli): give the #1169 analyze e2e the same 90s Windows budget as its sibling The first #1169 persist-meta case was still on a 60s spawn/it budget and was killed banner-only on windows-latest after the FTS warning fixture no longer failed the shard first. Co-authored-by: Cursor <cursoragent@cursor.com> * test(ci): reweight Windows shards after the FTS e2e grew Vendored-first HOME fixtures pushed fts-extension-e2e to ~6 minutes on windows-latest, so the old 146s weight packed it with skills-e2e and blew the 20-minute watchdog. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Copilot
|
87b91c821e
|
fix(lbug): add WAL checkpoint-threshold control (#1772)
* Initial plan * fix(analyze): add WAL auto-checkpoint CLI control and default-off behavior * test(analyze): share lbug auto-checkpoint parsing and align validation * fix(analyze): always enable lbug auto-checkpoint and expose threshold control * refactor(lbug): inline always-on auto-checkpoint constructor arg * fix(analyze): guide checkpoint-threshold on Ladybug WAL checkpoint IO failures * test(analyze): cover checkpoint IO guidance and add integration guard * fix(analyze): tighten checkpoint IO detection and remove test hook * fix(analyze): remove checkpoint test hook and tighten error matching * fix(analyze): rename to wal-checkpoint-threshold, raise default, add manual checkpoint driver with retry Address review feedback on PR #1772: - Rename CLI flag, env var, AnalyzeOptions field, recovery-hint tag, and parser/constants from lbug-* to engine-neutral wal-* (matches the existing WAL_RECOVERY_SUGGESTION / isWalCorruptionError convention). - Raise default threshold from -1 (Ladybug stock ~16 MiB) to 64 MiB so users on the default config no longer hit the original rename/remove race. - Align both READMEs to publish 67108864 (64 MiB) instead of 65536 (which would have made the crash more frequent). - Add wal-checkpoint-driver.ts: a periodic manual CHECKPOINT driver wrapped in a 3-attempt jittered retry (50/200/500 ms), driven from runFullAnalysis. Opt-out via GITNEXUS_WAL_MANUAL_CHECKPOINT=0. Moves the race window into a JS-controllable retry surface while keeping native auto-checkpoint on. - Move LBUG_CHECKPOINT_RENAME_RE / REMOVE_RE plus the predicate (renamed to isLbugCheckpointIoError) into lbug-config.ts alongside isWalCorruptionError. Predicate is now exported. Add a permissive fallback matcher and pin the matched Ladybug version in comments. - Warn instead of silently defaulting when GITNEXUS_WAL_CHECKPOINT_THRESHOLD is set to a non-empty unparseable value (closes the CLI-vs-env asymmetry). - Add a typed RecoveryHint string-literal union in cli-message.ts so future hint tags can't drift. - Add a real integration test under test/integration/ that triggers a Ladybug checkpoint IO failure via a pre-existing directory at the rename target (portable across platforms; no test-only injection hook). - Add small-disk / CI caveat (32 MiB secondary suggestion) to the recovery hint and README env-var rows. - Document CLI/env precedence in the analyze --help block. - Help placeholder: <value> -> <bytes>. - Rename analyze-lbug-auto-checkpoint.test.ts to use the new wal-* token. * chore(lbug): remove dead jitteredDelay helper and apply prettier - Drop unused `jitteredDelay` function flagged by CodeQL in PR #1772; the retry loop already inlines the same calculation with the injectable `randomImpl` so the helper was dead. Move the non-cryptographic-by-design comment next to the actual jitter site. - Apply `prettier --write` to wal-checkpoint-driver.ts and the new integration test to absorb the PR autofix bot's formatting findings. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Gergő Magyar <gergomagyar@icloud.com> Co-authored-by: Test <test@example.com> |