14 commits

Author	SHA1	Message	Date
mengkaka	4154b63131	feat(indexing): add Objective-C semantic indexing support (#3179) ... * docs: add Objective-C fork provider notes * feat(objective-c): add deterministic provider and grammar * feat(objective-c): finalize provider MVP * fix(objective-c): harden provider integration * fix(objective-c): normalize bare macro markers * docs(objective-c): integrate provider documentation * fix(objective-c): harden resolution and header classification * fix(objective-c): complete provider follow-ups * fix: address Objective-C review follow-ups * chore: format Objective-C grammar sources * fix(objective-c): harden review follow-ups * Address PR review feedback (#3179) Keep Objective-C chunking and macro recovery aligned with the grammar, and stop Community MEMBER_OF edges from leaking into symbol context. Co-authored-by: Cursor <cursoragent@cursor.com> * Address follow-up review on ObjC chunking and language fallback. Keep preprocessor directive text from changing file-scope brace depth, group real ivar nodes, skip header modifiers, and restore Rakefile/Gemfile detection through getLanguageFromFilename. Co-authored-by: Cursor <cursoragent@cursor.com> * Parse Objective-C headers with the objc grammar in embeddings. ensureAndParse and structural extraction now use the same content classifier as ingest, including method snippets from .h files, so Protocol/Category/Class chunks are not re-parsed as C++. Co-authored-by: Cursor <cursoragent@cursor.com> * Address PR review feedback (#3179) Keep file-scope macro elision off C line splices and @interface/@protocol/@implementation bodies, and attach ivar attributes to the following instance variable when chunking. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(bench): rebaseline Objective-C CSV emit * feat(objective-c): add workspace resolution and linear emit benches Plain .h files are classified as C++, so the ObjC pass could not resolve #import of those headers. Load a C/C#-style workspace once per pass, and keep protocol-candidate USES linear. Refs #3179 Co-authored-by: Cursor <cursoragent@cursor.com> * Address PR review feedback (#3179) - Compare LadybugDB labels() as a scalar when excluding Community MEMBER_OF edges. - Walk superclass members, skip file-static C sibling defs, and ignore comments in ObjC header/macro scans. Note: pre-existing failure in objective-c-provider integration (worker-pool ready timeout) not addressed by this PR. Co-authored-by: Cursor <cursoragent@cursor.com> * Address PR review feedback (#3179) Emit Objective-C declaration captures so compilation-unit siblings can share header/implementation bindings, and keep class vs protocol visibility groups distinct. Note: pre-existing failure in worker-pool startup (GITNEXUS_WORKER_READY_TIMEOUT_MS) not addressed by this PR. Co-authored-by: Cursor <cursoragent@cursor.com> * Address PR review feedback (#3179) Emit every comma-separated property/ivar declarator, and count @interface after a multiline block comment closes so in-declaration macros stay intact. Note: pre-existing failure in worker-pool startup (GITNEXUS_WORKER_READY_TIMEOUT_MS) not addressed by this PR. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: ximengkai <ximengkai@soyoung.com> Co-authored-by: Gergő Magyar <gergomagyar@icloud.com> Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>	2026-09-09 09:40:21 +00:00
Gergő Magyar	561f913a32	fix(embeddings): retry unparseable 200 responses and survive partial embedding failures (#2790) (#2795) ... * fix(embeddings): retry unparseable 200 responses and survive partial embedding failures (#2790) A long-running embedding job against an OpenAI-compatible endpoint could lose hours of work to a single transient glitch, then refuse to recover on the next run. Four defects compounded: 1. An HTTP 200 carrying a truncated or non-JSON body was never retried. `classifyOutcome` treats any 2xx as success, and the `resp.json()` parse ran after `resilientFetch` had already returned, so the parse failure surfaced as a terminal error. Measured: a 503 got 3 attempts, a garbage 200 got 1. The parse and the response-shape check now run inside the `fetchImpl` callback, so a bad body is classified as a retryable failure and gets the same backoff as a 5xx. This also stops a garbage 200 from calling the circuit breaker's `recordSuccess()`, which previously erased accumulated failures and meant an endpoint alternating 5xx and garbage-200 could never trip it. 2. One failed `embedBatch` sub-batch aborted the entire pipeline. Failures are now tolerated: the sub-batch's node ids are collected and all of their embedding rows are deleted, so those nodes hold zero rows and are re-embedded later. Deleting rather than keeping partial rows is deliberate — chunk arrays are flat over a 16-node batch and sliced by 8, so a node's chunks can straddle a sub-batch boundary, and surviving rows carry the current content hash. The hash maps collapse per-chunk rows last-row-wins, so a partially embedded node would read as fresh forever and never regenerate its missing chunks. A run that fails 5 sub-batches in a row still aborts, and rethrows the first error of the streak rather than the last: after 3 failures the circuit breaker opens, so later errors degrade into "circuit open, retry in 30s" while the first still names the real defect. 3. The Phase 5 `embeddingCount === 0` fail-fast could not tell "wrote nothing" from "could not ask" — the count query's catch was silent. The count is now tri-state and only a known zero after real work is fatal. A non-numeric count previously bypassed the gate entirely, because `Number()` returns NaN and `NaN === 0` is false, and then serialized as `embeddings: null`. An unverified count no longer certifies `capabilities.vectorSearch.status`. 4. `saveEmbeddingCheckpoint` wrote a completion-shaped meta: it advanced `lastCommit`, wrote the new `fileHashes` and cleared `incrementalInProgress`. The first checkpoint window fires before a single embedding exists, and on a full rebuild the graph is still in a staging database that a crash discards. The next run then diffed against the advanced hashes, saw no changes and preserved the old graph — the "skipping wipe" symptom in the report. It now re-reads meta and replaces only the checkpoint, matching what the server endpoint already did. A partially failed run keeps its checkpoint with the failed ids in `pendingNodeIds`, so the next plain `analyze` regenerates them through the existing resume path. Clearing it would have been silent data loss: a plain run derives `shouldGenerateEmbeddings: false` once embeddings exist, so the pipeline would never have run again. The old crash-and-abort self-healed only by accident, via the checkpoint its crash left behind. `gitnexus status` reports the index incomplete until the nodes recover, and `--drop-embeddings` still abandons them. `POST /api/embed` is the pipeline's other caller and was discarding the result, reporting "Embeddings complete" for a partial run. It now persists the pending ids and reports the run as failed with the underlying endpoint error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(embeddings): abort a run whose sub-batch failure ratio is too high (#2790) The consecutive-failure ceiling only catches a total outage, because any successful sub-batch resets it. An endpoint under load shedding that alternates success and failure never trips it, so the run walks the whole corpus, deletes every failed node's rows and exits 0 having dropped a large fraction of the index. The retained checkpoint made that visible in `gitnexus status`, but a run that drops a quarter of the corpus should tell the operator to fix their endpoint, not leave them to notice a status flag. Adds a cumulative guard: abort once more than 25% of attempted sub-batches have failed, evaluated as the run progresses and gated behind a floor of 20 attempted sub-batches. The shape follows Resilience4j's circuit breaker (failure rate plus a minimum-sample floor) because it is the only one of the surveyed designs that answers the small-repo case — a three node repo can fail one sub-batch and never accumulate enough sample for a ratio to mean anything. The rate sits below a live traffic breaker's 50% because a batch indexer's job is to index the whole corpus rather than serve degraded traffic, and above Hadoop's single-digit `failures.maxpercent` because tolerating transient hiccups is the point of the change this follows. The guard reuses the existing break-then-cleanup path, so the failed batch's DELETE still runs before the rethrow, and it wraps the retained first-error-of- streak rather than inventing a new one, so the message names both the ratio and the underlying endpoint failure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(server): record the embedding count after /api/embed so the next analyze cannot wipe it `POST /api/embed` generated embeddings and wrote them to the database but never wrote `stats.embeddings` into meta.json. Its checkpoint writer replaced only `embeddingCheckpoint`, and the finalize write folded in nothing else. So a repo embedded purely through the server kept whatever count the last CLI `analyze` stamped, which is 0 for a repo analyzed without embeddings. The next CLI run read `existingEmbeddingCount = 0`, `deriveEmbeddingMode` returned `shouldLoadCache: false`, and `gitnexus analyze --force` wiped the database with no cache load. Every server generated embedding was silently destroyed, with no warning — the user just lost semantic search. The route now measures the live count with the same query the CLI uses and folds it into both meta writes. The measurement is tri-state and deliberately never falls back to 0: an unverified count is written as absent rather than as zero, because a wrong-low value is exactly what arms the wipe. It is taken after `flushWAL()` and inside `withLbugDb`, so it describes durable rows and the connection is still open. A partial run records its honest count too, alongside the retained checkpoint, so the next CLI run preserves the partial index instead of discarding it. Found while working #2790; not part of that issue. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(embeddings): retry short 200 bodies and stop laundering body-phase timeouts Two gaps in the #2790 retry fix, both found by review. A 200 carrying `{"data": []}` or fewer vectors than inputs passed the in-`fetchImpl` shape check, because `every(isEmbeddingItem)` is vacuously true for an empty array. `resilientFetch` then classified it `success` and called `recordSuccess()`, erasing the outage signal, and the cardinality check in `httpEmbed` threw terminally one attempt later. That is exactly the pair of properties #2790 was filed about, still broken for this body shape — and worse than before the fix, since the pipeline now tolerates the error by deleting those nodes' rows instead of aborting loudly. The count check moves inside the retried callback; the outer one stays as a backstop. The `.json()` catch also swallowed every rejection, not just parse errors. `AbortSignal.any([caller, timeout])` is wired to the body stream, so a stalled body rejects with a DOMException — which, wrapped in a plain Error, defeated `classifyOutcome`'s terminal-network test. Measured: the same TimeoutError got 3 attempts and "unparseable response" when raised during the body read, but 1 attempt and "timed out after 180000ms" when raised by fetch itself, and three such sub-batches opened the process-global breaker that `recordNeutral()` exists to protect. Abort-like DOMExceptions are now re-raised unchanged. The dimension check stays outside the loop deliberately: it validates against `config.dimensions ?? DEFAULT_DIMS`, not the request-dimensions argument, and a width mismatch is a configuration error where retrying only triples latency and books failures against a healthy endpoint. Adds the negative assertion the review found missing: response body text must never reach the user-facing error string. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(embeddings): scale the sub-batch failure-ratio floor to the run The cumulative guard needed 20 attempted sub-batches before a failure rate could abort anything — roughly 160 chunks, or ~80 embeddable nodes at the default subBatchSize of 8. A 50-node repo whose endpoint sheds every other sub-batch fails half of them and still exits 0: the ratio guard is below its floor, and every intervening success resets the consecutive ceiling. The floor was a good choice for a first run over a small repo, where one failure out of one sub-batch is 100% and means nothing. The defect is that every resume run has that shape by construction — its node set is only the pending ids — so the guard was structurally off in the one run whose entire purpose is retrying against the endpoint that already failed. The floor is now sized to the run: clamp(ceil(totalNodes / 16), 5, 20). The lower bound keeps the case the flat floor protected; the upper bound preserves today's behavior above 320 nodes and avoids a proportional-only floor perversely weakening the guard at scale, where a sixteenth of a 20k-node repo would be 1250 sub-batches of damage before a rate could fire. Resilience4j can use a constant minimumNumberOfCalls because a breaker sits on an unbounded call stream; a batch indexer has a finite budget, so a constant can exceed the whole run. The ratio is still evaluated only inside the catch. That is already its local maximum — both counters have just incremented — so sampling more often would only ever observe lower ratios. Also: a failing cleanup DELETE no longer swallows the abort, which was discarding the retained first-error-of-the-streak that names the real endpoint fault; `ceilingError` is renamed `abortError` since it carries the ratio abort too; and three `{ error }` log keys become `{ err }` (#2114 — an arbitrary key serializes to `{}`, losing message and stack). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(analyze): one tri-state embedding counter, and stop partial runs wedging later runs The tri-state count doctrine this branch introduced was applied at two of its three CLI sites, and the two implementations that were meant to mirror each other had already drifted. `measurePersistedEmbeddingCount` moves to `core/embedding-count.ts` — beside `embedding-mode.ts`, with the same no-native-imports property, and outside `core/embeddings/` so the lazy-embeddings convention (#2370) still holds. All three call sites now share it. - The mid-run `onCheckpoint` counter ran the query bare. A throw there — DB busy, connection closed, read-only, the VECTOR DML lock (#2623) — rejected the callback out of `runEmbeddingPipeline` and killed the analyze before Phase 5 could apply the tri-state that exists for exactly this case. A non-numeric cell wrote `stats.embeddings: null` to disk mid-run. - Phase 5 used `?? 0` while the server used `?? Number.NaN`, under a comment asserting both measured the field the same way. `Number.isFinite(0)` is true, so a no-row answer became a *measured* zero and hard-failed a run whose embeddings had all persisted. - The unknown-count fallback read `existingMeta`, assigned once at run start, so it republished the pre-run figure over the fresher count the terminal checkpoint had already written. With a prior count of 0 that armed the wipe chain: hasExisting false, shouldLoadCache false, and the next --force discards live embeddings. It now re-reads the latest on-disk meta, and an unverifiable count retains a recovery marker instead of clearing it. A completed-but-partial run also planted a landmine. Its checkpoint is stamped with the run's embedding identity, so a later plain `gitnexus analyze` from a hook, a CI job, or a shell without GITNEXUS_EMBEDDING_URL resolved provider 'local' and threw before any phase ran — after an exit-0 run, where previously only a visible crash left that state. `--force` did not help: the resume gate inspected only `--drop-embeddings`. `RepoMeta.embeddingCheckpoint` gains `kind` to tell the two situations apart. An 'interrupted' marker (or one with no kind, so markers already on disk keep the stricter path) still fails closed — its nodes may be half-written, and resuming under a foreign model would mix vector spaces. A 'partial' marker names nodes the pipeline already deleted to zero rows, so nothing is at risk: an identity mismatch drops the pending set with a warning and continues. `--force` now discards a checkpoint, and `attempts` bounds the retry at EMBEDDING_RESUME_MAX_ATTEMPTS (3, matching the HTTP embedder's and the WAL driver's existing per-operation budgets) so a node the endpoint deterministically rejects converges instead of keeping the repo incomplete forever. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(server): close the SSE stream on terminal job status, not a progress phase A tolerated partial run reached SSE clients as a clean success — a regression in this branch's own claim that /api/embed reports a partial run as failed. The pipeline emits `phase:'ready'` unconditionally before returning, including when it dropped nodes. The route mapped that to `'complete'`, and `mountSSEProgress` treated a terminal-looking *progress phase* as terminal: write the event, `res.end()`, `unsubscribe()`. The route's own `updateJob({status:'failed'})` then fired into a stream with no listener, and the web app had already shown "ready". Before this branch the pipeline threw, which produced `phase:'error'` and did reach the client. Pollers on GET /api/embed/:jobId were unaffected, so the two consumers disagreed. Terminality is a property of the job, so the relay now asks the job. Remapping `ready` alone would have left the trap armed: the `error -> 'failed'` mapping has the identical shape and would emit `event: failed` with `error: undefined` before the catch block fills the message in. `ready` is additionally remapped to `finalizing` so a poller no longer sees `status:'analyzing'` next to `progress.phase:'complete'`. The single-terminal-event property (#2264) is preserved on both the clean and partial paths, and /api/analyze is unaffected — its terminal progress phase is 'done', never 'complete'. `AnalyzeJob` gains an optional `partial` payload so a client can tell a partial run from a total failure without a new status member; it is absent on every other job, so existing payloads stay byte-identical. Consuming it in gitnexus-web is left to that app's owner — today it renders both as the same red retry chip. `resolveEmbedRunOutcome` moves to `embed-run-outcome.ts` and `mountSSEProgress` to `sse-progress.ts`, both free of Express/LadybugDB/MCP imports, and the local count copy is replaced by the shared `core/embedding-count.ts`. Reaching three pure functions previously meant importing the whole server: measured at ~20s against a 30s test timeout, with one observed timeout failure. That file is now 1.6s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: document the partial embedding index and its recovery A run can now finish exit 0 with a partial embedding index, which neither operator doc described. GUARDRAILS' "Embeddings vanished after analyze" Sign keys its trigger on `stats.embeddings` being 0 and lists "the only ways to end up at zero". A partial run stamps an honest non-zero count and sets `embeddingCheckpoint`, so the operator's actual symptom is `incompleteReasons: ["embedding-checkpoint-pending"]` — a state that Sign cannot match. Adds a Sign for it and drops the exhaustive framing from the existing one. RUNBOOK gains the recovery path: a plain `gitnexus analyze` is correct and needs no flag, because a retained checkpoint forces generation for the pending nodes regardless of flags. Also corrects two stale claims — that `stats.embeddings` is always freshly measured (it can carry forward when the count query cannot answer, which is why `capabilities.vectorSearch.status` is the certified read), and that later analyzes must always pass `--embeddings` or lose their vectors, which contradicts Non-negotiable 5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(embeddings): one owner for the checkpoint record and the abort predicate Cleanup pass over the #2790 review fixes. No behavior change except where noted; the two exceptions are both cases where the code was lying to the operator or to the other half of itself. The previous pass extracted `core/embedding-count.ts` because two hand-copied bodies of "measure the embedding count" had drifted inside a single change. It then created a second pair of hand-copied publishers — of `RepoMeta.embeddingCheckpoint` — and those had drifted too: the CLI armed the attempt counter only after clearing its identity gate, the server derived it from the resumed marker alone. Only one of the two READERS implemented `kind` at all, so a 'partial' marker written by `gitnexus analyze` and resumed through POST /api/embed still hit the permanent wedge `kind` exists to remove. `core/embedding-checkpoint.ts` now owns the record: `checkpointKind` (the one home for absent-means-interrupted), the three minters, `nextAttemptCount`, and `decideEmbeddingResume`, which both gates route through. Five mint sites and two resume gates become one implementation each. `resilient-fetch.ts` exports `isTerminalNetworkError` and `classifyOutcome` calls it, replacing a caller-side copy of the same DOMException test whose docstring promised it "mirrors classifyOutcome exactly" — an invariant enforced by prose, where a divergence silently reverts body-phase timeouts to being retried three times and charged to the shared breaker. The ratio-guard floor now divides by the run's actual `subBatchSize` instead of a constant 16 that assumed the default of 8. At `subBatchSize: 32` the old formula demanded more sub-batches than the run contains, leaving the guard structurally off — the exact failure the scaled floor was introduced to fix, and sub-batch size is tuned mainly for the flaky endpoints it protects. Two operator-facing corrections: - The count-recovery marker was stamped `kind: 'partial'` with an empty pending set, so `gitnexus status` reported "N node(s) lost their embeddings" where N is zero. It gets its own kind and its own incomplete reason. - `decideEmbeddingResume` initially keyed its skip-the-identity-gate branch on an empty pending set, assuming that meant the count-recovery marker. It does not: `onCheckpoint` mints an 'interrupted' marker with no pending nodes after every post-window save. That silently cleared an interrupted marker under a foreign provider instead of failing closed. Keyed on `kind` now, with a regression test. Also: `isTerminalJobStatus` adopted at the seven sites that still hand-copied it, including the one gating the single-terminal-event emit; `mountSSEProgress` re-export dropped and `server-sse-payload.test.ts` repointed at the extracted module, which takes it from 24.60s to 0.408s — the test that motivated the extraction was still paying the cost it was meant to remove; the count-mismatch message and the SSE test harness deduplicated; per-batch error strings made lazy (~75k needless `new URL()` per large run); `retryable: true` dropped as a field that can never be false; ~110 lines of restated rationale reduced to pointers at their canonical home. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-02 20:43:59 +00:00
Parafee41	f45e89e6b6	fix(embeddings): make batch inserts retry-safe (#2453) ... * fix(embeddings): make batch inserts retry-safe * fix(types): cover optional transformers dependency * Fix embedding restore test expectation * test(embeddings): count checkpoint creates --------- Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>	2026-07-16 15:21:03 +01:00
Gergo Magyar	5869dde31d	fix(embeddings): make HTTP generation resumable (#2468)	2026-07-16 10:00:50 +00:00
Eva	711ff8721d	fix(embeddings): make HTTP generation resumable	2026-07-14 02:15:57 +07:00
Eva	554c5181cf	fix(embeddings): fall back to text-bearing file nodes	2026-07-14 01:40:47 +07:00
Gergő Magyar	905b7dfa21	feat(embeddings): compact, description-forward embedding text (#2333) (#2334)	2026-07-01 07:37:16 +01:00
Gergő Magyar	e46651e42c	fix(embeddings): create VECTOR index via conn.query, not the prepared path (#2114) ... `gitnexus analyze` silently failed to create the LadybugDB VECTOR/HNSW index because `CALL CREATE_VECTOR_INDEX(...)` was run through the prepared `conn.prepare()` path, which rejects multi-statement procedures — degrading semantic search to exact-scan. Route index creation through `conn.query()` via a new adapter-owned `createVectorIndex` (mirrors `createFTSIndex`), make the previously-swallowed error visible (`{ err }` logging), add an in-process idempotency cache, and add real-`@ladybugdb/core` regression coverage. Fixes #2114. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-10 07:59:37 +01:00
Gergő Magyar	2a5bbbeaae	fix: make extension installs offline-first (#1161) ... * feat(review): add PR reviewer swarm agents Seven read-only subagents coordinated by an orchestration skill for structured, evidence-grounded production-readiness PR reviews. Agents: facts-historian, branch-hygiene, risk-architect, test-ci-verifier, security-boundary, docs-dod, synthesis-critic. All use Read/Grep/Glob/Bash only — no edit tools. Skill invoked as /gitnexus-pr-swarm-review <PR>. * fix: patch vector extension and uncaughtException for review findings - Add { policy: 'auto' } to both loadVectorExtension() calls in embedding-pipeline.ts so analyze --embeddings auto-installs VECTOR - Add void to uncaughtException shutdown(1) call for Node v20+ safety - Re-add getExtensionInstallPolicy export + default change + 4 tests * fix(mcp,lbug): graceful shutdown exit codes + complete offline-first VECTOR policy Completes the two live issues PR #1161 only partially addressed. #1132 — MCP shutdown crash: SIGINT/SIGTERM were registered with `shutdown` directly, so Node passed the signal NAME string into process.exit(), crashing with ERR_INVALID_ARG_TYPE ('SIGTERM'). Map signals to numeric exit codes (SIGINT->130, SIGTERM->143) via a testable installSignalShutdown(); add an unref'd force-exit watchdog so a hung disconnect()/close() cannot wedge shutdown; and void the stdin/stdout handlers so event payloads never reach process.exit() as a non-number. #1153 — offline-first extension loading: - semanticSearch (a query/read path) no longer forces policy:'auto'; queries use load-only and never spawn a network INSTALL (extension.ladybugdb.com). - the analyze embedding WRITE path resolves the policy from GITNEXUS_LBUG_EXTENSION_INSTALL (honoring never/load-only/auto; default auto) instead of hard-forcing 'auto', so an offline/locked-down operator's override is respected (the regression that re-broke #1153 for the VECTOR path). - surface the active install policy in `gitnexus doctor` (was claimed but never delivered; also gives the previously-dead getExtensionInstallPolicy a caller). - emit an actionable message when VECTOR is unavailable. Tests: regression for the signal->numeric mapping (reproduces the signal-string crash condition) and for embedding install-policy resolution. tsc/prettier clean, eslint 0 errors, 55 unit tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(analyze): degrade gracefully when FTS extension is unavailable The load-only default made `gitnexus analyze` throw when the FTS extension was not pre-installed, breaking CI and offline use. Make the analyze write path opt into the `auto` install policy (LOAD-first then bounded INSTALL — symmetric with the VECTOR/embeddings path and the #726 contract) and degrade gracefully when the extension still cannot load: skip search-index creation, log a warning, and complete with a fully queryable graph (only full-text/BM25 search is disabled). `--repair-fts` still fails loudly. - Surface the degraded state instead of reporting healthy: AnalyzeResult.ftsSkipped, a persistent CLI summary warning, and meta.json capabilities.fts.status = "unavailable". - Skip the FTS-primitive integration tests when the extension is unavailable (shared skipUnlessFtsAvailable helper). - Add a unit test for the degradation branch; fix the existing full-analyze test mock that omitted loadFTSExtension. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(lbug): skip FTS-seeding suites when extension is unavailable The withTestLbugDB helper seeds FTS indexes in beforeAll via createFTSIndex, which throws when the optional FTS extension cannot load — failing the whole suite on machines where it is neither pre-installed nor installable (the macOS platform-sensitive CI runner). Probe the extension once (mirroring the analyze write path's `auto` policy), bypass FTS seeding when it is unavailable, and skip the suite's tests via beforeEach with a one-time warning so the skip is visible rather than a setup crash. Fixes the macOS failures in search-core, search-pool, local-backend-calltool, and staleness-and-stability. Suites still run normally where FTS is available. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-05-29 20:04:41 +01:00
Gergő Magyar	2a0c97c178	fix: add platform-aware semantic fallback (#1150) ... * fix: add platform-aware semantic fallback Make VECTOR an optional capability so Windows analysis remains stable while semantic embeddings can fall back to exact scan when native vector indexing is unavailable. Made-with: Cursor * fix: remove stale vector pool import Keep the merge with main lint-clean after VECTOR loading moved out of the read pool.	2026-04-28 12:21:25 +01:00
Gergő Magyar	ffa0510f9a	fix(lbug): prevent DuckDB extension install hangs (#1129) ... * fix(lbug): bound DuckDB extension install via ExtensionManager (closes #1128) `gitnexus analyze` could hang indefinitely (60% / 85% on Windows) when DuckDB's `INSTALL fts` or `INSTALL VECTOR` was unable to reach `extensions.duckdb.org`. The DuckDB driver's INSTALL is a synchronous network call, so any blocked egress would block the Node event loop forever. Replace the ad-hoc, in-process INSTALL/LOAD scattered across `lbug-adapter.ts` and `pool-adapter.ts` with a single `ExtensionManager` that owns the lifecycle of optional DuckDB extensions: * `LOAD` is always tried first — per-connection, idempotent, no network. * If `LOAD` fails and policy permits, INSTALL runs in a short-lived child Node process bounded by `GITNEXUS_LBUG_EXTENSION_INSTALL_TIMEOUT_MS` (default 15s). The parent loop keeps spinning; on timeout the child is killed with SIGKILL and the capability is flagged unavailable. * Capabilities and install attempts are cached per process, so a single bounded install per extension covers every subsequent call. Install policy is now an explicit, per-context decision: * `auto` (default for analyze) — try LOAD, fall back to bounded INSTALL. * `load-only` — used by `pool-adapter` (serve / MCP read paths) so user queries never block on a network install. * `never` — operator escape hatch for offline / airgapped environments. `createFTSIndex` and `createVectorIndex` now check the boolean return value before issuing the index DDL, so missing extensions degrade BM25 and semantic search gracefully without ever throwing during analyze. Tests: - New unit suite for `ExtensionManager` covering LOAD-first behavior, all three policies, install caching, observability, and warn dedup. - Existing vector-extension integration tests pass against the new boolean return type. - Existing embedding-pipeline mocks updated to return `true`. Docs: `gitnexus/README.md` documents `GITNEXUS_LBUG_EXTENSION_INSTALL` and `GITNEXUS_LBUG_EXTENSION_INSTALL_TIMEOUT_MS` with examples for offline and slow-network environments. Made-with: Cursor * fix(lbug): move DuckDB extension install child into script Keep the bounded out-of-process INSTALL behavior, but replace the inline child code with a stable packaged ESM script. This makes the child process directly runnable and gives debuggable stack traces without source-vs-dist branching or a runtime transpiler. Made-with: Cursor	2026-04-27 23:09:17 +01:00
evolution	2b7cff5fd2	feat(embeddings): structural chunking with data-driven CHUNKING_RULES dispatch (#987) ... * feat(embeddings): structural chunking with data-driven CHUNKING_RULES dispatch Replace hardcoded label comparisons with a CHUNKING_RULES lookup table that drives chunking strategy and text generation. Key changes: - Data-driven dispatch: CHUNKING_RULES table maps labels to chunking mode (ast-function / ast-declaration), prefix/suffix, field grouping, and structural text mode - Struct support: add Struct to AST declaration chunking with field grouping (same as Class) - Multi-chunk context: preceding chunk tail (prevTail) injected into embedding text for cross-chunk coherence - Version-gated hashes: EMBEDDING_TEXT_VERSION prefix in content hashes invalidates stale vectors when text template changes - Compact container context: first declaration line preserved in every structural chunk for identity * fix(embeddings): address PR review findings for CHUNKING_RULES refactor - Remove LABEL_ENUM from STRUCTURAL_LABELS to avoid wasted AST parses - Add maintenance note about extractStructuralNames and EMBEDDING_TEXT_VERSION - Clarify CHUNK_MODE_CHARACTER is a no-op in CHUNKING_RULES - Strengthen EMBEDDING_TEXT_VERSION test assertion to exact value --------- Co-authored-by: wangjichao <wangjichao@inke.cn>	2026-04-20 08:25:31 +01:00
evolution	02739085d2	feat(embeddings): AST-aware chunking with offset-based splitting (#889)	2026-04-16 22:55:04 +01:00
Copilot	1df79c2eab	fix: content-hash staleness detection for embeddings and vector index creation on zero-node path (#831) ... * Initial plan * fix: stale vectors preserved on content edits and vector index missing after zero-node run Issue 1: Add contentHash to EMBEDDING_SCHEMA and embedding pipeline. - contentHash column persisted per CodeEmbedding row - POST /api/embed queries nodeId+contentHash, compares per-node hash - Stale rows (hash mismatch) are DELETE'd before re-embedding - Legacy DBs without contentHash treated as stale (full re-embed) - loadCachedEmbeddings and run-analyze cache restore include contentHash Issue 2: createVectorIndex called unconditionally before zero-node early return. Regression tests: - contentHashForNode determinism and content-change detection - EMBEDDING_SCHEMA includes contentHash STRING column - Pipeline exports verified Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/1581c0c0-f359-4376-b47e-62d24a28fd2d Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com> * fix: use parameterized query for stale embedding DELETE, revert package-lock.json Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/1581c0c0-f359-4376-b47e-62d24a28fd2d Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com> * fix: address review feedback — config consistency, narrow catches, extract DB logic Bug #1: Use finalConfig consistently in contentHashForNode (line 224 was using raw `config` while line 307 used `finalConfig`). Cache precomputed hashes in filter phase to avoid double computation (Perf #5). Bug #2: Narrow catch in loadCachedEmbeddings to only fall back on column/table-missing errors. Rethrow transient/connection errors. Bug #3: Log non-trivial DELETE failures instead of silently swallowing. Arch Violation #3: Extract fetchExistingEmbeddingHashes from api.ts into lbug-adapter.ts. Server layer now calls a single adapter function instead of re-implementing the DB query logic with nested try-catch. Tests: Add config consistency test, note that fetchExistingEmbeddingHashes tests require native module (run in CI). Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/b8c4f6b0-4095-4507-a15d-d8469793efac Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com> * fix: narrow Column error match to 'contentHash' in lbug-adapter fallback checks Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/b8c4f6b0-4095-4507-a15d-d8469793efac Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com> * fix: address production-readiness review — eliminate competing state, use schema constants, hard-fail on stale DELETE, add incremental filter tests Gap A / Arch Violation 1: Remove duplicate vectorExtensionLoaded flag from embedding-pipeline.ts — delegate to lbug-adapter's loadVectorExtension() which owns the VECTOR extension lifecycle and resets on DB reconnect. Arch Violation 2: Replace all hardcoded 'CodeEmbedding' and 'code_embedding_idx' strings in embedding-pipeline.ts and run-analyze.ts with EMBEDDING_TABLE_NAME, EMBEDDING_INDEX_NAME, and CREATE_VECTOR_INDEX_QUERY imported from schema.ts. Add EMBEDDING_INDEX_NAME export to schema.ts. Gap B: Make DELETE failure for stale vectors a hard throw (not just a warning). Continuing after failed DELETE risks Kuzu vector-index corruption since the constraint requires DELETE-before-INSERT for vector-indexed properties. "not found" / "does not exist" errors are still safe to ignore. STALE_HASH_SENTINEL: Define a named constant in embedding types.ts for the empty-string sentinel convention. Used consistently in lbug-adapter.ts and run-analyze.ts so the invariant is self-documenting. Tests: Add comprehensive unit tests for the incremental filter logic with mocked embedder: - New node → embedded - Unchanged node (hash matches) → skipped - Stale node (hash mismatch) → DELETE + re-embed - STALE_HASH_SENTINEL → treated as stale - Zero nodes after filter → createVectorIndex still called - DELETE failure with non-trivial error → throws Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/b21edee7-c9c5-4742-947b-d0def4fb26aa Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com> * fix: tighten error classification — extract isMissingColumnOrTableError helper, remove broad pattern matching - Extract isMissingColumnOrTableError() helper in lbug-adapter for consistent schema-error detection (replaces duplicate inline checks) - Tighten 'contentHash' match: now requires 'property' AND 'contentHash' (Kuzu-specific pattern) instead of broad 'contentHash' substring - Tighten DELETE error check: only ignore 'does not exist' (Kuzu's actual message), not broad 'not found' which could mask connection errors - Fix test node ID/name/filePath consistency Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/b21edee7-c9c5-4742-947b-d0def4fb26aa Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com> * fix: CI failures and final review — move STALE_HASH_SENTINEL to schema, tighten error matching, fix test mocking, format - Move STALE_HASH_SENTINEL from embeddings/types.ts to lbug/schema.ts (fixes inverted layer dependency: lbug should not import from embeddings) - Tighten isMissingColumnOrTableError: replace broad msg.includes('not found') with /(table|column|property).*not found/i regex to avoid matching transient errors - Add vi.resetModules() in test beforeEach for explicit module isolation (fixes vi.doMock not intercepting loadVectorExtension in CI) - Skip precomputedHashes.set() on unchanged (return false) path - Run prettier on all 5 files flagged by CI format check Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/e20311fd-4361-47b4-a137-9adc3e533b35 * fix: address remaining review nits — rename precomputedHashes, generalize error matcher, revert package-lock - Rename precomputedHashes → computedStaleHashes (hashes are computed on-demand during filter, only cached for stale nodes being re-embedded) - Remove contentHash-specific clause from isMissingColumnOrTableError — the regex /(table|column|property).*not found/i already covers it - Revert package-lock.json ssh→https protocol change Agent-Logs-Url: https://github.com/abhigyanpatwari/GitNexus/sessions/e20311fd-4361-47b4-a137-9adc3e533b35 --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>	2026-04-15 11:20:48 +01:00