mirror of
https://github.com/abhigyanpatwari/GitNexus.git
synced 2026-08-28 05:25:25 +00:00
Some checks are pending
CodeQL / Analyze (javascript-typescript) (push) Waiting to run
CodeQL / Analyze (python) (push) Waiting to run
Gitleaks / gitleaks (push) Waiting to run
Publish / Classify release event (push) Waiting to run
Publish / RC guard (marker + release-PR skip) (push) Blocked by required conditions
Publish / ci (push) Blocked by required conditions
Publish / Publish to npm (push) Blocked by required conditions
Publish / Build & Push RC Docker images (push) Blocked by required conditions
Scorecard / Scorecard analysis (push) Waiting to run
Skill copy sync / shipped skills drift guard (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-cli) (push) Waiting to run
Trivy Image Scan / Trivy (gitnexus-web) (push) Waiting to run
* fix(embeddings): retry unparseable 200 responses and survive partial embedding failures (#2790) A long-running embedding job against an OpenAI-compatible endpoint could lose hours of work to a single transient glitch, then refuse to recover on the next run. Four defects compounded: 1. An HTTP 200 carrying a truncated or non-JSON body was never retried. `classifyOutcome` treats any 2xx as success, and the `resp.json()` parse ran after `resilientFetch` had already returned, so the parse failure surfaced as a terminal error. Measured: a 503 got 3 attempts, a garbage 200 got 1. The parse and the response-shape check now run inside the `fetchImpl` callback, so a bad body is classified as a retryable failure and gets the same backoff as a 5xx. This also stops a garbage 200 from calling the circuit breaker's `recordSuccess()`, which previously erased accumulated failures and meant an endpoint alternating 5xx and garbage-200 could never trip it. 2. One failed `embedBatch` sub-batch aborted the entire pipeline. Failures are now tolerated: the sub-batch's node ids are collected and all of their embedding rows are deleted, so those nodes hold zero rows and are re-embedded later. Deleting rather than keeping partial rows is deliberate — chunk arrays are flat over a 16-node batch and sliced by 8, so a node's chunks can straddle a sub-batch boundary, and surviving rows carry the current content hash. The hash maps collapse per-chunk rows last-row-wins, so a partially embedded node would read as fresh forever and never regenerate its missing chunks. A run that fails 5 sub-batches in a row still aborts, and rethrows the first error of the streak rather than the last: after 3 failures the circuit breaker opens, so later errors degrade into "circuit open, retry in 30s" while the first still names the real defect. 3. The Phase 5 `embeddingCount === 0` fail-fast could not tell "wrote nothing" from "could not ask" — the count query's catch was silent. The count is now tri-state and only a known zero after real work is fatal. A non-numeric count previously bypassed the gate entirely, because `Number()` returns NaN and `NaN === 0` is false, and then serialized as `embeddings: null`. An unverified count no longer certifies `capabilities.vectorSearch.status`. 4. `saveEmbeddingCheckpoint` wrote a completion-shaped meta: it advanced `lastCommit`, wrote the new `fileHashes` and cleared `incrementalInProgress`. The first checkpoint window fires before a single embedding exists, and on a full rebuild the graph is still in a staging database that a crash discards. The next run then diffed against the advanced hashes, saw no changes and preserved the old graph — the "skipping wipe" symptom in the report. It now re-reads meta and replaces only the checkpoint, matching what the server endpoint already did. A partially failed run keeps its checkpoint with the failed ids in `pendingNodeIds`, so the next plain `analyze` regenerates them through the existing resume path. Clearing it would have been silent data loss: a plain run derives `shouldGenerateEmbeddings: false` once embeddings exist, so the pipeline would never have run again. The old crash-and-abort self-healed only by accident, via the checkpoint its crash left behind. `gitnexus status` reports the index incomplete until the nodes recover, and `--drop-embeddings` still abandons them. `POST /api/embed` is the pipeline's other caller and was discarding the result, reporting "Embeddings complete" for a partial run. It now persists the pending ids and reports the run as failed with the underlying endpoint error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(embeddings): abort a run whose sub-batch failure ratio is too high (#2790) The consecutive-failure ceiling only catches a total outage, because any successful sub-batch resets it. An endpoint under load shedding that alternates success and failure never trips it, so the run walks the whole corpus, deletes every failed node's rows and exits 0 having dropped a large fraction of the index. The retained checkpoint made that visible in `gitnexus status`, but a run that drops a quarter of the corpus should tell the operator to fix their endpoint, not leave them to notice a status flag. Adds a cumulative guard: abort once more than 25% of attempted sub-batches have failed, evaluated as the run progresses and gated behind a floor of 20 attempted sub-batches. The shape follows Resilience4j's circuit breaker (failure rate plus a minimum-sample floor) because it is the only one of the surveyed designs that answers the small-repo case — a three node repo can fail one sub-batch and never accumulate enough sample for a ratio to mean anything. The rate sits below a live traffic breaker's 50% because a batch indexer's job is to index the whole corpus rather than serve degraded traffic, and above Hadoop's single-digit `failures.maxpercent` because tolerating transient hiccups is the point of the change this follows. The guard reuses the existing break-then-cleanup path, so the failed batch's DELETE still runs before the rethrow, and it wraps the retained first-error-of- streak rather than inventing a new one, so the message names both the ratio and the underlying endpoint failure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(server): record the embedding count after /api/embed so the next analyze cannot wipe it `POST /api/embed` generated embeddings and wrote them to the database but never wrote `stats.embeddings` into meta.json. Its checkpoint writer replaced only `embeddingCheckpoint`, and the finalize write folded in nothing else. So a repo embedded purely through the server kept whatever count the last CLI `analyze` stamped, which is 0 for a repo analyzed without embeddings. The next CLI run read `existingEmbeddingCount = 0`, `deriveEmbeddingMode` returned `shouldLoadCache: false`, and `gitnexus analyze --force` wiped the database with no cache load. Every server generated embedding was silently destroyed, with no warning — the user just lost semantic search. The route now measures the live count with the same query the CLI uses and folds it into both meta writes. The measurement is tri-state and deliberately never falls back to 0: an unverified count is written as absent rather than as zero, because a wrong-low value is exactly what arms the wipe. It is taken after `flushWAL()` and inside `withLbugDb`, so it describes durable rows and the connection is still open. A partial run records its honest count too, alongside the retained checkpoint, so the next CLI run preserves the partial index instead of discarding it. Found while working #2790; not part of that issue. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(embeddings): retry short 200 bodies and stop laundering body-phase timeouts Two gaps in the #2790 retry fix, both found by review. A 200 carrying `{"data": []}` or fewer vectors than inputs passed the in-`fetchImpl` shape check, because `every(isEmbeddingItem)` is vacuously true for an empty array. `resilientFetch` then classified it `success` and called `recordSuccess()`, erasing the outage signal, and the cardinality check in `httpEmbed` threw terminally one attempt later. That is exactly the pair of properties #2790 was filed about, still broken for this body shape — and worse than before the fix, since the pipeline now tolerates the error by deleting those nodes' rows instead of aborting loudly. The count check moves inside the retried callback; the outer one stays as a backstop. The `.json()` catch also swallowed every rejection, not just parse errors. `AbortSignal.any([caller, timeout])` is wired to the body stream, so a stalled body rejects with a DOMException — which, wrapped in a plain Error, defeated `classifyOutcome`'s terminal-network test. Measured: the same TimeoutError got 3 attempts and "unparseable response" when raised during the body read, but 1 attempt and "timed out after 180000ms" when raised by fetch itself, and three such sub-batches opened the process-global breaker that `recordNeutral()` exists to protect. Abort-like DOMExceptions are now re-raised unchanged. The dimension check stays outside the loop deliberately: it validates against `config.dimensions ?? DEFAULT_DIMS`, not the request-dimensions argument, and a width mismatch is a configuration error where retrying only triples latency and books failures against a healthy endpoint. Adds the negative assertion the review found missing: response body text must never reach the user-facing error string. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(embeddings): scale the sub-batch failure-ratio floor to the run The cumulative guard needed 20 attempted sub-batches before a failure rate could abort anything — roughly 160 chunks, or ~80 embeddable nodes at the default subBatchSize of 8. A 50-node repo whose endpoint sheds every other sub-batch fails half of them and still exits 0: the ratio guard is below its floor, and every intervening success resets the consecutive ceiling. The floor was a good choice for a first run over a small repo, where one failure out of one sub-batch is 100% and means nothing. The defect is that every resume run has that shape by construction — its node set is only the pending ids — so the guard was structurally off in the one run whose entire purpose is retrying against the endpoint that already failed. The floor is now sized to the run: clamp(ceil(totalNodes / 16), 5, 20). The lower bound keeps the case the flat floor protected; the upper bound preserves today's behavior above 320 nodes and avoids a proportional-only floor perversely weakening the guard at scale, where a sixteenth of a 20k-node repo would be 1250 sub-batches of damage before a rate could fire. Resilience4j can use a constant minimumNumberOfCalls because a breaker sits on an unbounded call stream; a batch indexer has a finite budget, so a constant can exceed the whole run. The ratio is still evaluated only inside the catch. That is already its local maximum — both counters have just incremented — so sampling more often would only ever observe lower ratios. Also: a failing cleanup DELETE no longer swallows the abort, which was discarding the retained first-error-of-the-streak that names the real endpoint fault; `ceilingError` is renamed `abortError` since it carries the ratio abort too; and three `{ error }` log keys become `{ err }` (#2114 — an arbitrary key serializes to `{}`, losing message and stack). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(analyze): one tri-state embedding counter, and stop partial runs wedging later runs The tri-state count doctrine this branch introduced was applied at two of its three CLI sites, and the two implementations that were meant to mirror each other had already drifted. `measurePersistedEmbeddingCount` moves to `core/embedding-count.ts` — beside `embedding-mode.ts`, with the same no-native-imports property, and outside `core/embeddings/` so the lazy-embeddings convention (#2370) still holds. All three call sites now share it. - The mid-run `onCheckpoint` counter ran the query bare. A throw there — DB busy, connection closed, read-only, the VECTOR DML lock (#2623) — rejected the callback out of `runEmbeddingPipeline` and killed the analyze before Phase 5 could apply the tri-state that exists for exactly this case. A non-numeric cell wrote `stats.embeddings: null` to disk mid-run. - Phase 5 used `?? 0` while the server used `?? Number.NaN`, under a comment asserting both measured the field the same way. `Number.isFinite(0)` is true, so a no-row answer became a *measured* zero and hard-failed a run whose embeddings had all persisted. - The unknown-count fallback read `existingMeta`, assigned once at run start, so it republished the pre-run figure over the fresher count the terminal checkpoint had already written. With a prior count of 0 that armed the wipe chain: hasExisting false, shouldLoadCache false, and the next --force discards live embeddings. It now re-reads the latest on-disk meta, and an unverifiable count retains a recovery marker instead of clearing it. A completed-but-partial run also planted a landmine. Its checkpoint is stamped with the run's embedding identity, so a later plain `gitnexus analyze` from a hook, a CI job, or a shell without GITNEXUS_EMBEDDING_URL resolved provider 'local' and threw before any phase ran — after an exit-0 run, where previously only a visible crash left that state. `--force` did not help: the resume gate inspected only `--drop-embeddings`. `RepoMeta.embeddingCheckpoint` gains `kind` to tell the two situations apart. An 'interrupted' marker (or one with no kind, so markers already on disk keep the stricter path) still fails closed — its nodes may be half-written, and resuming under a foreign model would mix vector spaces. A 'partial' marker names nodes the pipeline already deleted to zero rows, so nothing is at risk: an identity mismatch drops the pending set with a warning and continues. `--force` now discards a checkpoint, and `attempts` bounds the retry at EMBEDDING_RESUME_MAX_ATTEMPTS (3, matching the HTTP embedder's and the WAL driver's existing per-operation budgets) so a node the endpoint deterministically rejects converges instead of keeping the repo incomplete forever. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(server): close the SSE stream on terminal job status, not a progress phase A tolerated partial run reached SSE clients as a clean success — a regression in this branch's own claim that /api/embed reports a partial run as failed. The pipeline emits `phase:'ready'` unconditionally before returning, including when it dropped nodes. The route mapped that to `'complete'`, and `mountSSEProgress` treated a terminal-looking *progress phase* as terminal: write the event, `res.end()`, `unsubscribe()`. The route's own `updateJob({status:'failed'})` then fired into a stream with no listener, and the web app had already shown "ready". Before this branch the pipeline threw, which produced `phase:'error'` and did reach the client. Pollers on GET /api/embed/:jobId were unaffected, so the two consumers disagreed. Terminality is a property of the job, so the relay now asks the job. Remapping `ready` alone would have left the trap armed: the `error -> 'failed'` mapping has the identical shape and would emit `event: failed` with `error: undefined` before the catch block fills the message in. `ready` is additionally remapped to `finalizing` so a poller no longer sees `status:'analyzing'` next to `progress.phase:'complete'`. The single-terminal-event property (#2264) is preserved on both the clean and partial paths, and /api/analyze is unaffected — its terminal progress phase is 'done', never 'complete'. `AnalyzeJob` gains an optional `partial` payload so a client can tell a partial run from a total failure without a new status member; it is absent on every other job, so existing payloads stay byte-identical. Consuming it in gitnexus-web is left to that app's owner — today it renders both as the same red retry chip. `resolveEmbedRunOutcome` moves to `embed-run-outcome.ts` and `mountSSEProgress` to `sse-progress.ts`, both free of Express/LadybugDB/MCP imports, and the local count copy is replaced by the shared `core/embedding-count.ts`. Reaching three pure functions previously meant importing the whole server: measured at ~20s against a 30s test timeout, with one observed timeout failure. That file is now 1.6s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: document the partial embedding index and its recovery A run can now finish exit 0 with a partial embedding index, which neither operator doc described. GUARDRAILS' "Embeddings vanished after analyze" Sign keys its trigger on `stats.embeddings` being 0 and lists "the only ways to end up at zero". A partial run stamps an honest non-zero count and sets `embeddingCheckpoint`, so the operator's actual symptom is `incompleteReasons: ["embedding-checkpoint-pending"]` — a state that Sign cannot match. Adds a Sign for it and drops the exhaustive framing from the existing one. RUNBOOK gains the recovery path: a plain `gitnexus analyze` is correct and needs no flag, because a retained checkpoint forces generation for the pending nodes regardless of flags. Also corrects two stale claims — that `stats.embeddings` is always freshly measured (it can carry forward when the count query cannot answer, which is why `capabilities.vectorSearch.status` is the certified read), and that later analyzes must always pass `--embeddings` or lose their vectors, which contradicts Non-negotiable 5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(embeddings): one owner for the checkpoint record and the abort predicate Cleanup pass over the #2790 review fixes. No behavior change except where noted; the two exceptions are both cases where the code was lying to the operator or to the other half of itself. The previous pass extracted `core/embedding-count.ts` because two hand-copied bodies of "measure the embedding count" had drifted inside a single change. It then created a second pair of hand-copied publishers — of `RepoMeta.embeddingCheckpoint` — and those had drifted too: the CLI armed the attempt counter only after clearing its identity gate, the server derived it from the resumed marker alone. Only one of the two READERS implemented `kind` at all, so a 'partial' marker written by `gitnexus analyze` and resumed through POST /api/embed still hit the permanent wedge `kind` exists to remove. `core/embedding-checkpoint.ts` now owns the record: `checkpointKind` (the one home for absent-means-interrupted), the three minters, `nextAttemptCount`, and `decideEmbeddingResume`, which both gates route through. Five mint sites and two resume gates become one implementation each. `resilient-fetch.ts` exports `isTerminalNetworkError` and `classifyOutcome` calls it, replacing a caller-side copy of the same DOMException test whose docstring promised it "mirrors classifyOutcome exactly" — an invariant enforced by prose, where a divergence silently reverts body-phase timeouts to being retried three times and charged to the shared breaker. The ratio-guard floor now divides by the run's actual `subBatchSize` instead of a constant 16 that assumed the default of 8. At `subBatchSize: 32` the old formula demanded more sub-batches than the run contains, leaving the guard structurally off — the exact failure the scaled floor was introduced to fix, and sub-batch size is tuned mainly for the flaky endpoints it protects. Two operator-facing corrections: - The count-recovery marker was stamped `kind: 'partial'` with an empty pending set, so `gitnexus status` reported "N node(s) lost their embeddings" where N is zero. It gets its own kind and its own incomplete reason. - `decideEmbeddingResume` initially keyed its skip-the-identity-gate branch on an empty pending set, assuming that meant the count-recovery marker. It does not: `onCheckpoint` mints an 'interrupted' marker with no pending nodes after every post-window save. That silently cleared an interrupted marker under a foreign provider instead of failing closed. Keyed on `kind` now, with a regression test. Also: `isTerminalJobStatus` adopted at the seven sites that still hand-copied it, including the one gating the single-terminal-event emit; `mountSSEProgress` re-export dropped and `server-sse-payload.test.ts` repointed at the extracted module, which takes it from 24.60s to 0.408s — the test that motivated the extraction was still paying the cost it was meant to remove; the count-mismatch message and the SSE test harness deduplicated; per-batch error strings made lazy (~75k needless `new URL()` per large run); `retryable: true` dropped as a field that can never be false; ~110 lines of restated rationale reduced to pointers at their canonical home. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
687 lines
28 KiB
TypeScript
687 lines
28 KiB
TypeScript
import fs from 'node:fs/promises';
|
|
import os from 'node:os';
|
|
import path from 'node:path';
|
|
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
|
|
import { JobManager } from '../../src/server/analyze-job.js';
|
|
import {
|
|
startSSEHarness,
|
|
terminalFrame,
|
|
terminalFrameCount,
|
|
type SSEHarness,
|
|
} from '../helpers/sse-harness.js';
|
|
import {
|
|
resolveEmbedRunOutcome,
|
|
withMeasuredEmbeddingCount,
|
|
type EmbeddingRunResult,
|
|
} from '../../src/server/embed-run-outcome.js';
|
|
import { mintInterruptedCheckpoint } from '../../src/core/embedding-checkpoint.js';
|
|
import {
|
|
measurePersistedEmbeddingCount,
|
|
persistedEmbeddingCountOrUndefined,
|
|
} from '../../src/core/embedding-count.js';
|
|
import { loadMeta, saveMeta, type RepoMeta } from '../../src/storage/repo-manager.js';
|
|
import { deriveEmbeddingMode } from '../../src/core/embedding-mode.js';
|
|
|
|
/**
|
|
* NOTHING in this file imports `src/server/api.ts` for behavior. That module
|
|
* pulls Express, cors, the LadybugDB native adapter and the whole MCP wiring:
|
|
* reaching three pure helpers through it cost one 30s TIMEOUT and ~20s/~22s on
|
|
* the runs that passed, against a 30s `testTimeout` (#2790 review, finding 9).
|
|
* The helpers now live in `src/server/{sse-progress,embed-run-outcome}.ts` and
|
|
* `src/core/embedding-{count,checkpoint}.ts`, none of which import a database
|
|
* or a server.
|
|
*/
|
|
|
|
describe('analyze API logic', () => {
|
|
let manager: JobManager;
|
|
|
|
beforeEach(() => {
|
|
manager = new JobManager();
|
|
});
|
|
|
|
afterEach(() => {
|
|
manager.dispose();
|
|
});
|
|
|
|
it('creates a job and returns 202 shape', () => {
|
|
const job = manager.createJob({ repoUrl: 'https://github.com/user/repo' });
|
|
const response = { jobId: job.id, status: job.status };
|
|
expect(response.jobId).toBeTruthy();
|
|
expect(response.status).toBe('queued');
|
|
});
|
|
|
|
it('rejects when job already active for different repo', () => {
|
|
const job1 = manager.createJob({ repoUrl: 'https://github.com/user/repo1' });
|
|
manager.updateJob(job1.id, { status: 'analyzing' });
|
|
expect(() => manager.createJob({ repoUrl: 'https://github.com/user/repo2' })).toThrow(
|
|
/already in progress/,
|
|
);
|
|
});
|
|
|
|
it('returns existing job for same repo URL', () => {
|
|
const job1 = manager.createJob({ repoUrl: 'https://github.com/user/repo' });
|
|
manager.updateJob(job1.id, { status: 'analyzing' });
|
|
const job2 = manager.createJob({ repoUrl: 'https://github.com/user/repo' });
|
|
expect(job2.id).toBe(job1.id);
|
|
});
|
|
|
|
it('SSE progress listener receives all events including terminal', () => {
|
|
const job = manager.createJob({ repoUrl: 'https://github.com/user/sse-test' });
|
|
const events: Array<{ phase: string; percent: number }> = [];
|
|
const unsub = manager.onProgress(job.id, (progress) => {
|
|
events.push({ phase: progress.phase, percent: progress.percent });
|
|
});
|
|
|
|
manager.updateJob(job.id, {
|
|
status: 'analyzing',
|
|
progress: { phase: 'parsing', percent: 30, message: 'Parsing' },
|
|
});
|
|
manager.updateJob(job.id, {
|
|
progress: { phase: 'calls', percent: 50, message: 'Tracing calls' },
|
|
});
|
|
manager.updateJob(job.id, { status: 'complete', repoName: 'sse-test' });
|
|
|
|
unsub();
|
|
|
|
expect(events).toEqual([
|
|
{ phase: 'parsing', percent: 30 },
|
|
{ phase: 'calls', percent: 50 },
|
|
{ phase: 'complete', percent: 100 },
|
|
]);
|
|
});
|
|
});
|
|
|
|
const IDENTITY = { model: 'test-model', dimensions: 384, provider: 'local' };
|
|
const CLEAN_RUN: EmbeddingRunResult = {
|
|
nodesProcessed: 412,
|
|
chunksProcessed: 900,
|
|
failedNodeIds: [],
|
|
};
|
|
/** Progress figures an in-flight checkpoint records. */
|
|
const PROGRESS = { nodesProcessed: 4, totalNodes: 12, chunksProcessed: 9 };
|
|
|
|
/**
|
|
* ── #2790: an SSE client must not be told a partial run succeeded ──────────
|
|
*
|
|
* `runEmbeddingPipeline` emits `phase: 'ready'` / 100% UNCONDITIONALLY before
|
|
* returning — including when it dropped nodes to endpoint failures — and
|
|
* /api/embed relayed that as a progress phase before it had measured anything
|
|
* or decided the outcome. The relay treated a progress PHASE STRING of
|
|
* 'complete'/'failed' as terminal, so it wrote `event: complete` with
|
|
* `error: undefined`, called `res.end()` and unsubscribed; the route's later
|
|
* `updateJob({status:'failed'})` went into a stream with no listener. The web
|
|
* client fired `onComplete` and showed "ready" while a `GET /api/embed/:jobId`
|
|
* poller saw `failed` — the two consumers of one job disagreeing about whether
|
|
* the data is complete, and a regression against the pre-#2790 behavior where
|
|
* the pipeline threw and the client received the failure.
|
|
*
|
|
* These tests drive the REAL relay over a REAL HTTP server (same harness as
|
|
* server-sse-payload.test.ts) and subscribe BEFORE the misleading event is
|
|
* emitted — subscribing after it is exactly why the previous version of this
|
|
* suite passed while the bug was live.
|
|
*/
|
|
describe('mountSSEProgress terminality (#2790)', () => {
|
|
let harness: SSEHarness;
|
|
let manager: JobManager;
|
|
let baseUrl = '';
|
|
|
|
beforeEach(async () => {
|
|
// Mirrors both production mounts in createServer().
|
|
harness = await startSSEHarness('/api/embed/:jobId/progress');
|
|
manager = harness.manager;
|
|
baseUrl = harness.baseUrl;
|
|
});
|
|
|
|
afterEach(() => harness.close());
|
|
|
|
it('a partial run reaches the client as a failure, not a success', async () => {
|
|
const job = manager.createJob({ repoPath: '/ws/embed-partial' });
|
|
manager.updateJob(job.id, {
|
|
repoName: 'embed-partial',
|
|
status: 'analyzing',
|
|
progress: { phase: 'embedding', percent: 40, message: 'Embedding nodes (40%)...' },
|
|
});
|
|
|
|
// The client is connected and listening BEFORE anything terminal-looking is
|
|
// emitted. `fetch` resolves once headers arrive, and the handler subscribes
|
|
// synchronously before that (see server-sse-payload.test.ts).
|
|
const response = await fetch(`${baseUrl}/api/embed/${job.id}/progress`);
|
|
|
|
// A progress event that CLAIMS to be terminal. Production now maps the
|
|
// pipeline's `ready` to 'finalizing' instead, but a phase string must not be
|
|
// able to end the stream no matter who sends it — that is the invariant.
|
|
manager.updateJob(job.id, {
|
|
progress: { phase: 'complete', percent: 100, message: 'Embeddings complete' },
|
|
});
|
|
|
|
// Only now does the route learn the run dropped nodes.
|
|
const outcome = resolveEmbedRunOutcome(IDENTITY, {
|
|
nodesProcessed: 10,
|
|
chunksProcessed: 24,
|
|
failedNodeIds: ['node-a', 'node-b'],
|
|
});
|
|
manager.updateJob(job.id, {
|
|
status: 'failed',
|
|
error: outcome.error,
|
|
partial: outcome.partial,
|
|
progress: { phase: 'failed', percent: 100, message: String(outcome.error) },
|
|
});
|
|
|
|
const body = await response.text();
|
|
|
|
expect(body).not.toContain('event: complete');
|
|
expect(terminalFrameCount(body)).toBe(1);
|
|
expect(terminalFrame(body, 'failed')).toMatchObject({
|
|
repoName: 'embed-partial',
|
|
repoPath: '/ws/embed-partial',
|
|
error: expect.stringContaining('finished partially') as unknown as string,
|
|
// The distinction a UI needs to offer "retry 2 nodes" instead of a bare
|
|
// red chip — carried without adding a `status` union member.
|
|
partial: { kind: 'embedding-partial', pendingNodeCount: 2, nodesProcessed: 10 },
|
|
});
|
|
});
|
|
|
|
it('a clean run produces exactly one terminal complete event', async () => {
|
|
const job = manager.createJob({ repoPath: '/ws/embed-clean' });
|
|
manager.updateJob(job.id, {
|
|
repoName: 'embed-clean',
|
|
status: 'analyzing',
|
|
progress: { phase: 'embedding', percent: 40, message: 'Embedding nodes (40%)...' },
|
|
});
|
|
|
|
const response = await fetch(`${baseUrl}/api/embed/${job.id}/progress`);
|
|
|
|
// What the route actually emits between the pipeline returning and the
|
|
// outcome being known.
|
|
manager.updateJob(job.id, {
|
|
progress: { phase: 'finalizing', percent: 100, message: 'Finalizing embeddings...' },
|
|
});
|
|
manager.updateJob(job.id, {
|
|
status: 'complete',
|
|
progress: { phase: 'complete', percent: 100, message: 'Embeddings complete' },
|
|
});
|
|
|
|
const body = await response.text();
|
|
|
|
expect(body).not.toContain('event: failed');
|
|
// Exactly one — the status update carries a `progress` too, and #2264's
|
|
// single-emit rule is what keeps that from double-writing the terminal frame.
|
|
expect(terminalFrameCount(body)).toBe(1);
|
|
expect(terminalFrame(body, 'complete')).toEqual({
|
|
repoName: 'embed-clean',
|
|
repoPath: '/ws/embed-clean',
|
|
});
|
|
// The 'finalizing' frame was relayed as ordinary progress, not swallowed.
|
|
expect(body).toContain('"phase":"finalizing"');
|
|
});
|
|
|
|
it('the analyze path still closes on its own terminal update', async () => {
|
|
// /api/analyze mounts the same relay. Its worker reports phases like
|
|
// 'parsing' and 'done' (never 'complete'), so the fix must not leave that
|
|
// stream open — it closes when the job's STATUS becomes terminal.
|
|
const job = manager.createJob({ repoPath: '/ws/reels' });
|
|
manager.updateJob(job.id, {
|
|
status: 'analyzing',
|
|
progress: { phase: 'parsing', percent: 30, message: 'Parsing' },
|
|
});
|
|
|
|
const response = await fetch(`${baseUrl}/api/embed/${job.id}/progress`);
|
|
|
|
manager.updateJob(job.id, {
|
|
progress: { phase: 'done', percent: 100, message: 'Done' },
|
|
});
|
|
manager.updateJob(job.id, { status: 'complete', repoName: 'reels' });
|
|
|
|
const body = await response.text();
|
|
|
|
expect(terminalFrameCount(body)).toBe(1);
|
|
expect(terminalFrame(body, 'complete')).toEqual({ repoName: 'reels', repoPath: '/ws/reels' });
|
|
});
|
|
|
|
it('a job that finished before the client connected replays its outcome', async () => {
|
|
const job = manager.createJob({ repoPath: '/ws/embed-late' });
|
|
const outcome = resolveEmbedRunOutcome(IDENTITY, {
|
|
nodesProcessed: 3,
|
|
chunksProcessed: 9,
|
|
failedNodeIds: ['node-a'],
|
|
});
|
|
manager.updateJob(job.id, {
|
|
status: 'failed',
|
|
repoName: 'embed-late',
|
|
error: outcome.error,
|
|
partial: outcome.partial,
|
|
});
|
|
|
|
const body = await (await fetch(`${baseUrl}/api/embed/${job.id}/progress`)).text();
|
|
|
|
expect(terminalFrameCount(body)).toBe(1);
|
|
expect(terminalFrame(body, 'failed')).toMatchObject({
|
|
error: expect.stringContaining('finished partially') as unknown as string,
|
|
partial: { kind: 'embedding-partial', pendingNodeCount: 1, nodesProcessed: 3 },
|
|
});
|
|
});
|
|
});
|
|
|
|
/**
|
|
* ── #2790: POST /api/embed must not report unqualified success ─────────
|
|
*
|
|
* The pipeline no longer throws when a sub-batch loses its endpoint — it
|
|
* deletes the affected nodes' rows and names them in `failedNodeIds`. The route
|
|
* discarded that receipt: it cleared `embeddingCheckpoint` and marked the job
|
|
* 'complete', so a partial run looked identical to a clean one and the dropped
|
|
* nodes were never retried (pre-#2790 the pipeline threw and the catch marked
|
|
* the job failed).
|
|
*/
|
|
describe('resolveEmbedRunOutcome (#2790)', () => {
|
|
it('clears the checkpoint and reports no error on a clean, measured run', () => {
|
|
const outcome = resolveEmbedRunOutcome(IDENTITY, CLEAN_RUN, { measuredEmbeddings: 412 });
|
|
expect(outcome.checkpoint).toBeUndefined();
|
|
expect(outcome.error).toBeUndefined();
|
|
expect(outcome.partial).toBeUndefined();
|
|
});
|
|
|
|
it('retains the checkpoint with the dropped ids and reports an error on a partial run', () => {
|
|
const outcome = resolveEmbedRunOutcome(IDENTITY, {
|
|
nodesProcessed: 10,
|
|
chunksProcessed: 24,
|
|
failedNodeIds: ['node-a', 'node-b'],
|
|
});
|
|
// The record of what failed survives — this is the pending set the next
|
|
// run's `forceReembedNodeIds` re-embeds.
|
|
expect(outcome.checkpoint).toMatchObject({
|
|
pendingNodeIds: ['node-a', 'node-b'],
|
|
nodesProcessed: 10,
|
|
totalNodes: 12,
|
|
chunksProcessed: 24,
|
|
model: 'test-model',
|
|
dimensions: 384,
|
|
provider: 'local',
|
|
// The run COMPLETED: these nodes provably hold zero rows, so a later
|
|
// identity mismatch may drop the set with a warning instead of wedging
|
|
// every subsequent run (repo-manager.ts).
|
|
kind: 'partial',
|
|
});
|
|
expect(outcome.error).toMatch(/2 node\(s\)/);
|
|
expect(outcome.partial).toEqual({
|
|
kind: 'embedding-partial',
|
|
pendingNodeCount: 2,
|
|
nodesProcessed: 10,
|
|
});
|
|
});
|
|
|
|
it('stamps no attempt count on a fresh partial run', () => {
|
|
const outcome = resolveEmbedRunOutcome(
|
|
IDENTITY,
|
|
{ nodesProcessed: 10, chunksProcessed: 24, failedNodeIds: ['node-a'] },
|
|
// Resumed from an in-flight marker, not a partial one.
|
|
{ resumedFrom: mintInterruptedCheckpoint(IDENTITY, PROGRESS, ['node-a']) },
|
|
);
|
|
expect(outcome.checkpoint).toMatchObject({ kind: 'partial' });
|
|
expect(outcome.checkpoint?.attempts).toBeUndefined();
|
|
});
|
|
|
|
it('advances the attempt count only when a resumed pending node fails again', () => {
|
|
const resumedFrom: RepoMeta['embeddingCheckpoint'] = {
|
|
at: new Date(0).toISOString(),
|
|
nodesProcessed: 10,
|
|
totalNodes: 12,
|
|
chunksProcessed: 24,
|
|
...IDENTITY,
|
|
kind: 'partial',
|
|
attempts: 1,
|
|
pendingNodeIds: ['node-a', 'node-b'],
|
|
};
|
|
|
|
// Same node failed again → the retry is not converging; the budget advances.
|
|
expect(
|
|
resolveEmbedRunOutcome(
|
|
IDENTITY,
|
|
{ nodesProcessed: 11, chunksProcessed: 26, failedNodeIds: ['node-a'] },
|
|
{ resumedFrom },
|
|
).checkpoint,
|
|
).toMatchObject({ kind: 'partial', attempts: 2 });
|
|
|
|
// The resumed set cleared and DIFFERENT nodes were lost → a fresh partial,
|
|
// so the budget resets. The bound exists for a node the endpoint rejects
|
|
// deterministically, not for an endpoint that is merely flaky.
|
|
expect(
|
|
resolveEmbedRunOutcome(
|
|
IDENTITY,
|
|
{ nodesProcessed: 11, chunksProcessed: 26, failedNodeIds: ['node-z'] },
|
|
{ resumedFrom },
|
|
).checkpoint?.attempts,
|
|
).toBeUndefined();
|
|
});
|
|
});
|
|
|
|
describe('the mid-run marker /api/embed writes (mintInterruptedCheckpoint, #2790)', () => {
|
|
it('stamps interrupted, so resume regenerates a possibly half-written window', () => {
|
|
const checkpoint = mintInterruptedCheckpoint(IDENTITY, PROGRESS, ['node-a', 'node-b']);
|
|
expect(checkpoint).toMatchObject({
|
|
kind: 'interrupted',
|
|
nodesProcessed: 4,
|
|
totalNodes: 12,
|
|
chunksProcessed: 9,
|
|
model: 'test-model',
|
|
dimensions: 384,
|
|
provider: 'local',
|
|
pendingNodeIds: ['node-a', 'node-b'],
|
|
});
|
|
// `attempts` bounds retries of a 'partial' set; an in-flight marker has no
|
|
// such budget because its rows may exist.
|
|
expect(checkpoint.attempts).toBeUndefined();
|
|
});
|
|
});
|
|
|
|
/**
|
|
* ── The /api/embed count omission (silent embedding loss) ──────────────
|
|
*
|
|
* The route generated embeddings and wrote `embeddingCheckpoint`, but never
|
|
* `stats.embeddings`. A repo embedded purely through the server therefore kept
|
|
* whatever count the last CLI `analyze` stamped — `0` for a repo analyzed
|
|
* without embeddings. The next CLI run reads that as `existingEmbeddingCount`,
|
|
* `deriveEmbeddingMode` sees `hasExisting: false` → `shouldLoadCache: false`,
|
|
* and `gitnexus analyze --force` wipes the database with no cache load: every
|
|
* server-generated embedding is destroyed with no warning.
|
|
*
|
|
* The route body is an inline closure inside `createServer`, so its finalize
|
|
* sequence is replayed here over the SAME helpers the route calls, with real
|
|
* meta.json I/O and the real `deriveEmbeddingMode`. The consequence is what
|
|
* these tests pin, not the field.
|
|
*/
|
|
describe('POST /api/embed records the embedding count it measured', () => {
|
|
let metaDir: string;
|
|
let seeded: RepoMeta;
|
|
|
|
beforeEach(async () => {
|
|
metaDir = await fs.mkdtemp(path.join(os.tmpdir(), 'gn-embed-count-'));
|
|
});
|
|
|
|
afterEach(async () => {
|
|
await fs.rm(metaDir, { recursive: true, force: true });
|
|
});
|
|
|
|
/** What a CLI `analyze` (plus any mid-run checkpoint) leaves on disk. */
|
|
const seedMeta = async (
|
|
embeddings: number | undefined,
|
|
embeddingCheckpoint?: RepoMeta['embeddingCheckpoint'],
|
|
): Promise<void> => {
|
|
seeded = {
|
|
repoPath: '/repo/embed-count',
|
|
lastCommit: 'abc123',
|
|
indexedAt: new Date(0).toISOString(),
|
|
stats: { nodes: 500, ...(embeddings === undefined ? {} : { embeddings }) },
|
|
embeddingCheckpoint,
|
|
};
|
|
await saveMeta(metaDir, seeded);
|
|
};
|
|
|
|
const rowsWith = (cnt: unknown) => async () => [{ cnt } as Record<string, unknown>];
|
|
|
|
/** The route's finalize sequence: measure → re-read meta → resolve → write. */
|
|
const finalizeEmbedRun = async (
|
|
runQuery: (cypher: string) => Promise<Array<Record<string, unknown>> | undefined>,
|
|
pipelineResult: EmbeddingRunResult,
|
|
): Promise<RepoMeta | null> => {
|
|
const measured = await measurePersistedEmbeddingCount(runQuery);
|
|
const finalMeta = (await loadMeta(metaDir)) ?? seeded;
|
|
const outcome = resolveEmbedRunOutcome(IDENTITY, pipelineResult, {
|
|
measuredEmbeddings: persistedEmbeddingCountOrUndefined(measured),
|
|
onDisk: finalMeta,
|
|
});
|
|
await saveMeta(
|
|
metaDir,
|
|
withMeasuredEmbeddingCount(
|
|
{ ...finalMeta, embeddingCheckpoint: outcome.checkpoint },
|
|
measured,
|
|
),
|
|
);
|
|
return loadMeta(metaDir);
|
|
};
|
|
|
|
const embeddingCountOf = (meta: RepoMeta | null): number => meta?.stats?.embeddings ?? 0;
|
|
|
|
it('writes the measured count into meta on a clean run, without disturbing the other stats', async () => {
|
|
await seedMeta(0);
|
|
const asked: string[] = [];
|
|
const written = await finalizeEmbedRun(async (cypher) => {
|
|
asked.push(cypher);
|
|
return [{ cnt: 412 }];
|
|
}, CLEAN_RUN);
|
|
|
|
expect(written).toMatchObject({ stats: { nodes: 500, embeddings: 412 } });
|
|
// A clean, MEASURED run clears the checkpoint (#2790 contract).
|
|
expect(written?.embeddingCheckpoint).toBeUndefined();
|
|
// Measured, not restated: the count comes from the live embedding table.
|
|
expect(asked).toEqual([expect.stringMatching(/MATCH \(e:\w+\) RETURN count\(e\) AS cnt/)]);
|
|
});
|
|
|
|
it('is what makes the next CLI run preserve instead of wipe', async () => {
|
|
await seedMeta(0);
|
|
|
|
// Pre-fix state: the server embedded 412 nodes but meta still says 0.
|
|
const stale = embeddingCountOf(await loadMeta(metaDir));
|
|
expect(stale).toBe(0);
|
|
expect(deriveEmbeddingMode({ force: true }, stale)).toMatchObject({
|
|
// `--force` rebuilds without loading the embedding cache → the 412
|
|
// server-generated vectors are destroyed.
|
|
shouldLoadCache: false,
|
|
preserveExistingEmbeddings: false,
|
|
});
|
|
|
|
const written = await finalizeEmbedRun(rowsWith(412), CLEAN_RUN);
|
|
const honest = embeddingCountOf(written);
|
|
expect(honest).toBe(412);
|
|
|
|
// Post-fix: `--force` loads the cache and regenerates on top of it rather
|
|
// than discarding the index. (`preserveExistingEmbeddings` is false here by
|
|
// design — `--force` upgrades to `forceRegenerateEmbeddings`; the wipe
|
|
// protection is `shouldLoadCache`.)
|
|
expect(deriveEmbeddingMode({ force: true }, honest)).toMatchObject({
|
|
shouldLoadCache: true,
|
|
forceRegenerateEmbeddings: true,
|
|
});
|
|
// A routine `analyze` preserves them outright.
|
|
expect(deriveEmbeddingMode({}, honest)).toMatchObject({
|
|
shouldLoadCache: true,
|
|
preserveExistingEmbeddings: true,
|
|
});
|
|
});
|
|
|
|
it('treats an unanswerable count query as unknown rather than 0', async () => {
|
|
// The query throws for reasons unrelated to how many rows were written.
|
|
await expect(
|
|
measurePersistedEmbeddingCount(async () => {
|
|
throw new Error('Connection closed');
|
|
}),
|
|
).resolves.toMatchObject({ kind: 'unknown', reason: 'Connection closed' });
|
|
// No row / no cell: an empty table would still answer with a 0.
|
|
await expect(measurePersistedEmbeddingCount(async () => [])).resolves.toMatchObject({
|
|
kind: 'unknown',
|
|
});
|
|
await expect(measurePersistedEmbeddingCount(async () => undefined)).resolves.toMatchObject({
|
|
kind: 'unknown',
|
|
});
|
|
// Non-numeric cell — same class of unknown.
|
|
await expect(measurePersistedEmbeddingCount(rowsWith('many'))).resolves.toMatchObject({
|
|
kind: 'unknown',
|
|
});
|
|
// A real zero is still a real answer.
|
|
await expect(measurePersistedEmbeddingCount(rowsWith(0))).resolves.toEqual({
|
|
kind: 'measured',
|
|
count: 0,
|
|
});
|
|
});
|
|
|
|
it('leaves the previous count alone when the measurement fails, never writing a fabricated 0', async () => {
|
|
await seedMeta(137);
|
|
const written = await finalizeEmbedRun(async () => {
|
|
throw new Error('Connection closed');
|
|
}, CLEAN_RUN);
|
|
|
|
expect(written).toMatchObject({ stats: { embeddings: 137 } });
|
|
// The dangerous direction is wrong-LOW: a fabricated 0 here would arm the
|
|
// wipe the test above describes.
|
|
expect(deriveEmbeddingMode({ force: true }, embeddingCountOf(written))).toMatchObject({
|
|
shouldLoadCache: true,
|
|
});
|
|
});
|
|
|
|
it('keeps the recovery marker when a clean run cannot verify its own count', async () => {
|
|
// The state that arms the silent wipe: meta records 0 embeddings (a repo
|
|
// analyzed without them, embedded through the server), the run succeeded,
|
|
// and the count query cannot answer — so no honest count can be stamped.
|
|
const midRunMarker = mintInterruptedCheckpoint(IDENTITY, PROGRESS, ['node-a']);
|
|
await seedMeta(0, midRunMarker);
|
|
|
|
const written = await finalizeEmbedRun(async () => {
|
|
throw new Error('Connection closed');
|
|
}, CLEAN_RUN);
|
|
|
|
// No fabricated value: neither a 0 nor a NaN/null lands in meta.
|
|
expect(written).toMatchObject({ stats: { nodes: 500, embeddings: 0 } });
|
|
// …and the marker this run wrote SURVIVES, so something on disk still
|
|
// records that embeddings were produced. Clearing it here would leave the
|
|
// index with zero evidence of its own embeddings.
|
|
expect(written?.embeddingCheckpoint).toMatchObject({
|
|
kind: 'interrupted',
|
|
pendingNodeIds: ['node-a'],
|
|
});
|
|
});
|
|
|
|
it('still clears the marker on an unverifiable run once meta records embeddings', async () => {
|
|
// Same unmeasurable run, but the recorded count already proves the index is
|
|
// accounted for — nothing needs preserving, so the clean-run contract wins.
|
|
await seedMeta(412, mintInterruptedCheckpoint(IDENTITY, PROGRESS, ['node-a']));
|
|
|
|
const written = await finalizeEmbedRun(async () => {
|
|
throw new Error('Connection closed');
|
|
}, CLEAN_RUN);
|
|
|
|
expect(written).toMatchObject({ stats: { embeddings: 412 } });
|
|
expect(written?.embeddingCheckpoint).toBeUndefined();
|
|
});
|
|
|
|
it('records the honest count on a partial run, alongside the pending checkpoint', async () => {
|
|
await seedMeta(0);
|
|
const written = await finalizeEmbedRun(rowsWith(300), {
|
|
nodesProcessed: 300,
|
|
chunksProcessed: 700,
|
|
failedNodeIds: ['node-a', 'node-b'],
|
|
});
|
|
|
|
// A partial index that is honest about itself survives the next run: the
|
|
// count keeps `--force` from wiping it, the checkpoint re-embeds the rest.
|
|
expect(written).toMatchObject({
|
|
stats: { embeddings: 300 },
|
|
embeddingCheckpoint: {
|
|
pendingNodeIds: ['node-a', 'node-b'],
|
|
nodesProcessed: 300,
|
|
kind: 'partial',
|
|
},
|
|
});
|
|
expect(deriveEmbeddingMode({ force: true }, embeddingCountOf(written))).toMatchObject({
|
|
shouldLoadCache: true,
|
|
});
|
|
});
|
|
});
|
|
|
|
/**
|
|
* Wiring guard for the route. Everything the helpers DECIDE is pinned
|
|
* behaviorally above; what remains is that the inline route closure inside
|
|
* `createServer` still asks them — the helper being right while the call site
|
|
* keeps writing `embeddingCheckpoint: undefined` is exactly the regression
|
|
* #2790 is about, and that closure cannot be reached without booting a server
|
|
* over a real repo + LadybugDB + embedding endpoint. Static-analysis layer of
|
|
* last resort, same precedent as api-readonly-wiring.test.ts.
|
|
*/
|
|
describe('POST /api/embed route wiring (#2790)', () => {
|
|
const readSource = () =>
|
|
fs.readFile(path.join(__dirname, '..', '..', 'src', 'server', 'api.ts'), 'utf-8');
|
|
|
|
/**
|
|
* The body of the route's `withLbugDb` callback — everything that may only
|
|
* run while the database connection is open. Sliced rather than matched with
|
|
* a character-distance regex so a comment edit cannot silently un-assert it.
|
|
*/
|
|
const insideWithLbugDb = (source: string): string => {
|
|
const start = source.indexOf('await withLbugDb(lbugPath, async () => {');
|
|
const end = source.indexOf('\n });', start);
|
|
expect(start).toBeGreaterThan(-1);
|
|
expect(end).toBeGreaterThan(start);
|
|
return source.slice(start, end);
|
|
};
|
|
|
|
it('feeds the pipeline result through resolveEmbedRunOutcome into the finalize write', async () => {
|
|
const source = await readSource();
|
|
// The result is captured, not discarded…
|
|
expect(source).toContain('const pipelineResult = await runEmbeddingPipeline(');
|
|
// …handed to the helper with the finalize context…
|
|
expect(source).toMatch(
|
|
/resolveEmbedRunOutcome\(\s*embeddingIdentity,\s*pipelineResult,\s*finalizeContext,\s*\)/,
|
|
);
|
|
// …and its checkpoint is what the finalize meta write persists (pre-fix: a
|
|
// hardcoded `embeddingCheckpoint: undefined`).
|
|
expect(source).toContain('embeddingCheckpoint: outcome.checkpoint');
|
|
expect(source).toContain('partialRunError = outcome.error;');
|
|
// A partial run does not reach `status: 'complete'`, and carries its detail.
|
|
expect(source).toMatch(
|
|
/partialRunError === undefined[\s\S]{0,400}status: 'complete'[\s\S]{0,600}status: 'failed'/,
|
|
);
|
|
expect(source).toContain('partial: partialRunDetail,');
|
|
});
|
|
|
|
it('measures after the WAL flush, inside withLbugDb, and folds the result into the write', async () => {
|
|
const source = await readSource();
|
|
const region = insideWithLbugDb(source);
|
|
// Inside the open connection — this is the route's only chance to stamp
|
|
// `stats.embeddings`, and the next CLI run's preserve-or-wipe decision
|
|
// hangs on it.
|
|
expect(region).toContain('const measuredEmbeddings = await countPersistedEmbeddings();');
|
|
expect(region).toContain('await saveMeta(entry.storagePath, embeddingMeta);');
|
|
// Ordering, without brittle character spans: flush → measure → decide →
|
|
// write. Counting before the flush would describe rows still in the WAL.
|
|
const flushed = region.lastIndexOf('await flushWAL();');
|
|
const measured = region.indexOf('const measuredEmbeddings = await countPersistedEmbeddings();');
|
|
const decided = region.indexOf('const outcome = resolveEmbedRunOutcome(');
|
|
const folded = region.indexOf('embeddingMeta = withMeasuredEmbeddingCount(', measured);
|
|
expect(flushed).toBeLessThan(measured);
|
|
expect(measured).toBeLessThan(decided);
|
|
expect(decided).toBeLessThan(folded);
|
|
expect(region.slice(folded)).toContain('measuredEmbeddings,');
|
|
});
|
|
|
|
it('measures in the post-flush checkpoint callback and nowhere else in the pipeline options', async () => {
|
|
const source = await readSource();
|
|
expect(source).toContain(
|
|
'await saveEmbeddingCheckpoint(checkpoint, [], await countPersistedEmbeddings());',
|
|
);
|
|
// The window-start callback fires before any row exists — it must pass no
|
|
// count rather than restate a stale one.
|
|
expect(source).toMatch(
|
|
/onCheckpointWindowStart: async \(\{ nodeIds, \.\.\.checkpoint \}\) => \{\s*await saveEmbeddingCheckpoint\(checkpoint, nodeIds\);\s*\},/,
|
|
);
|
|
});
|
|
|
|
it('resolves a found checkpoint through the shared resume decision', async () => {
|
|
const source = await readSource();
|
|
const region = insideWithLbugDb(source);
|
|
// The route asks the SAME decider the CLI does, instead of hard-throwing on
|
|
// any identity mismatch and ignoring `attempts` — the disagreement that let
|
|
// a CLI-written `'partial'` marker wedge every later `POST /api/embed`.
|
|
expect(region).toMatch(/decideEmbeddingResume\(priorCheckpoint, embeddingIdentity\)/);
|
|
// Every action is routed: abort fails the run, abandon warns and proceeds
|
|
// with an empty pending set, resume hands the decision's ids to the pipeline.
|
|
expect(region).toContain("if (resume?.action === 'abort') throw new Error(resume.error);");
|
|
expect(region).toMatch(/resume\?\.action === 'resume'\s*\?\s*resume\.pendingNodeIds/);
|
|
// No second copy of the gate: the route no longer authors its own message.
|
|
expect(region).not.toContain('Cannot resume embedding checkpoint:');
|
|
});
|
|
|
|
it('never maps the pipeline ready phase to a phase a client can read as terminal', async () => {
|
|
const source = await readSource();
|
|
// `ready` fires unconditionally before the route knows the outcome (#2790).
|
|
expect(source).toMatch(/p\.phase === 'ready'\s*\?\s*'finalizing'/);
|
|
expect(source).not.toMatch(/p\.phase === 'ready' \? 'complete'/);
|
|
});
|
|
});
|