mirror of
https://github.com/supermemoryai/supermemory.git
synced 2026-11-04 07:30:17 +00:00
Merge c5db753fe3 into 59b148e5b2
This commit is contained in:
commit
5fc7af2972
8 changed files with 609 additions and 8 deletions
|
|
@ -1 +1,9 @@
|
|||
## supermemory Browser Extension
|
||||
## supermemory Browser Extension
|
||||
|
||||
### Hybrid recall evaluation
|
||||
|
||||
Auto-recall proof (legacy `memory`-only formatting vs hybrid `memory || chunk`):
|
||||
|
||||
- **Doc:** [docs/hybrid-recall-eval.md](./docs/hybrid-recall-eval.md)
|
||||
- **Report:** `bun run eval:hybrid-recall`
|
||||
- **Tests:** `bun test utils/hybrid-recall.eval.test.ts`
|
||||
|
|
|
|||
177
apps/browser-extension/docs/hybrid-recall-eval.md
Normal file
177
apps/browser-extension/docs/hybrid-recall-eval.md
Normal file
|
|
@ -0,0 +1,177 @@
|
|||
# Hybrid recall — evaluation & proof
|
||||
|
||||
This doc explains **what we claim**, **how we prove it**, and **how to re-run the proof** for the browser-extension auto-recall change on branch `feat/extension-hybrid-recall`.
|
||||
|
||||
## Claim (precise)
|
||||
|
||||
Auto-recall in the extension used to:
|
||||
|
||||
1. Call `/v4/search` **without** `searchMode` (API default = `"memories"`), and
|
||||
2. Format each hit as `` `${result.memory}` `` only.
|
||||
|
||||
When the API returns a **document chunk** (`chunk` set, `memory` absent), the injected “Included Memories” prompt got the literal string `undefined` and **lost** that document context.
|
||||
|
||||
**After this change**, auto-recall:
|
||||
|
||||
1. Requests `searchMode: "hybrid"` (memories **and** document chunks), and
|
||||
2. Formats each hit as `memory || chunk`, skipping empty hits and renumbering contiguously.
|
||||
|
||||
We claim an improvement in **context the extension injects into the LLM**, not an improvement to Supermemory’s private ranking/embedding engine.
|
||||
|
||||
| In scope | Out of scope |
|
||||
|----------|----------------|
|
||||
| Request body uses hybrid | Server ranking / fusion quality |
|
||||
| Prompt lines no longer contain `undefined` | MemoryBench provider scores |
|
||||
| Chunk-only API hits become usable prompt text | Changing how memories are stored |
|
||||
| Empty hits don’t invent numbered lines | End-to-end browser chrome E2E |
|
||||
|
||||
## How the proof works
|
||||
|
||||
We run an **offline A/B** on fixed gold fixtures that look like hybrid `/v4/search` payloads:
|
||||
|
||||
```
|
||||
same fixture results
|
||||
│
|
||||
├─► legacy formatter → metrics L
|
||||
│ (memory field only — old background.ts)
|
||||
│
|
||||
└─► new formatter → metrics N
|
||||
(memory || chunk — search-request.ts)
|
||||
```
|
||||
|
||||
Fixtures live in `utils/hybrid-recall.eval.ts` (`HYBRID_RECALL_FIXTURES`):
|
||||
|
||||
| Fixture id | What it models |
|
||||
|------------|----------------|
|
||||
| `mixed-memory-and-chunk` | Typical mix: extracted facts + one RAG chunk |
|
||||
| `chunk-starvation-shape` | Memory-heavy list with a single surviving doc chunk |
|
||||
| `chunk-only-page` | Document-only hits (no extracted memories) |
|
||||
| `empty-and-whitespace-noise` | Garbage / blank hits must not pollute the prompt |
|
||||
|
||||
### Metrics
|
||||
|
||||
| Metric | Meaning |
|
||||
|--------|---------|
|
||||
| **undefined lines** | Prompt lines matching `\bundefined\b` (legacy bug signature) |
|
||||
| **chunk-only recall** | How many fixture strings that exist *only* as `chunk` appear in the prompt |
|
||||
| **expected recall** | How many gold substrings appear in the prompt |
|
||||
| **usable lines** | Non-empty lines that are not `undefined` |
|
||||
| **requestUsesHybrid** | `buildSearchMemoriesBody` sets `searchMode: "hybrid"` |
|
||||
|
||||
### Pass gates (`assertHybridRecallPass`)
|
||||
|
||||
The harness **fails CI-style** unless all of these hold:
|
||||
|
||||
1. Search body uses `searchMode: "hybrid"`.
|
||||
2. New formatter: **0** undefined lines.
|
||||
3. Legacy formatter: **> 0** undefined lines (fixtures still prove the gap).
|
||||
4. New recovers **all** chunk-only gold strings; legacy recovers **none**.
|
||||
5. New recovers **all** expected strings and **beats** legacy expected recovery.
|
||||
|
||||
That combination both **shows the bug** and **shows the fix**.
|
||||
|
||||
## Run the proof
|
||||
|
||||
From `apps/browser-extension`:
|
||||
|
||||
```bash
|
||||
# Human-readable before/after report (exit 1 on FAIL)
|
||||
bun run eval:hybrid-recall
|
||||
|
||||
# Same gates as automated tests (+ unit coverage)
|
||||
bun test utils/hybrid-recall.eval.test.ts utils/search-request.test.ts
|
||||
```
|
||||
|
||||
Equivalent:
|
||||
|
||||
```bash
|
||||
bun utils/hybrid-recall.eval.ts
|
||||
bun test
|
||||
```
|
||||
|
||||
## Latest local result (re-run to refresh)
|
||||
|
||||
Captured with `bun run eval:hybrid-recall`:
|
||||
|
||||
```
|
||||
fixtures: 4
|
||||
request searchMode hybrid: true
|
||||
|
||||
undefined lines legacy=6 next=0
|
||||
chunk-only recall legacy=0/4 next=4/4
|
||||
expected recall legacy=6/10 next=10/10
|
||||
|
||||
PASS: new path recovers all fixture context; legacy gap confirmed.
|
||||
```
|
||||
|
||||
### Example before / after (one fixture)
|
||||
|
||||
**Fixture:** `mixed-memory-and-chunk`
|
||||
|
||||
| | Legacy | Next |
|
||||
|---|--------|------|
|
||||
| Line 1 | Prefers dark mode | Prefers dark mode |
|
||||
| Line 2 | Uses Biome for formatting | Uses Biome for formatting |
|
||||
| Line 3 | **`undefined`** | Deploy checklist: run migrations before restarting the API |
|
||||
|
||||
The third hit is chunk-only. Legacy injects noise; next injects the document text the model can use.
|
||||
|
||||
### Chunk-only page (strongest gap)
|
||||
|
||||
**Fixture:** `chunk-only-page`
|
||||
|
||||
| Legacy | Next |
|
||||
|--------|------|
|
||||
| `1. undefined` | Incident runbook… |
|
||||
| `2. undefined` | Rollback: redeploy… |
|
||||
|
||||
Legacy usable context: **0**. Next: **2/2** document lines.
|
||||
|
||||
## How to present this (PR / review)
|
||||
|
||||
Suggested PR section:
|
||||
|
||||
```markdown
|
||||
## Proof
|
||||
|
||||
Offline A/B eval: `bun run eval:hybrid-recall` in `apps/browser-extension`.
|
||||
|
||||
| Metric | Legacy | Next |
|
||||
|--------|--------|------|
|
||||
| undefined prompt lines | 6 | 0 |
|
||||
| chunk-only texts recovered | 0/4 | 4/4 |
|
||||
| expected texts recovered | 6/10 | 10/10 |
|
||||
|
||||
Doc: `apps/browser-extension/docs/hybrid-recall-eval.md`
|
||||
Harness: `utils/hybrid-recall.eval.ts`
|
||||
```
|
||||
|
||||
Attach the full CLI report as a PR comment or screenshot of the test output.
|
||||
|
||||
## Manual smoke (optional, not automated)
|
||||
|
||||
Use when you want confidence against a **live** API (requires your account + loaded extension build):
|
||||
|
||||
1. Load the extension from a local `bun run build` / `bun run dev` output.
|
||||
2. In one Supermemory project, save (a) a short preference fact and (b) a multi-sentence note/doc whose wording won’t be extracted as a memory yet (or search in a way that returns chunks).
|
||||
3. On ChatGPT/Claude/T3, type a query that should hit both.
|
||||
4. Open **Included Memories**.
|
||||
5. Confirm: no `undefined` rows; document wording appears when the hit is a chunk.
|
||||
|
||||
This smoke check is complementary. The offline harness is the **regression-proof** gate; smoke validates wiring to a real `/v4/search`.
|
||||
|
||||
## What this does *not* replace
|
||||
|
||||
- **MemoryBench** (`supermemoryai/memorybench`) scores providers’ ingest→search→answer quality. It does not exercise this Chrome content-script formatter.
|
||||
- Server-side hybrid ranking quirks (e.g. memory-heavy containers starving chunks — see supermemory#1398) are unchanged by this PR. We only ensure that **when** a chunk is returned, the extension uses it.
|
||||
|
||||
## File map
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `utils/search-request.ts` | `searchMode: "hybrid"` + `formatSearchHitsForPrompt` |
|
||||
| `entrypoints/background.ts` | Wires formatter into `GET_RELATED_MEMORIES` |
|
||||
| `utils/hybrid-recall.eval.ts` | Fixtures, legacy vs next scoring, CLI report |
|
||||
| `utils/hybrid-recall.eval.test.ts` | Hard pass gates in `bun test` |
|
||||
| `utils/search-request.test.ts` | Unit tests for body + hit formatting |
|
||||
| `docs/hybrid-recall-eval.md` | This document |
|
||||
|
|
@ -9,6 +9,7 @@ import {
|
|||
MESSAGE_TYPES,
|
||||
POSTHOG_EVENT_KEY,
|
||||
} from "../utils/constants"
|
||||
import { formatSearchHitsForPrompt } from "../utils/search-request"
|
||||
import { trackEvent } from "../utils/posthog"
|
||||
import { captureTwitterTokens } from "../utils/twitter-auth"
|
||||
import {
|
||||
|
|
@ -226,12 +227,9 @@ export default defineBackground(() => {
|
|||
|
||||
const responseData = await searchMemories(data, containerTag)
|
||||
const response = responseData as {
|
||||
results?: Array<{ memory?: string }>
|
||||
results?: Array<{ memory?: string; chunk?: string }>
|
||||
}
|
||||
const memories: string[] = []
|
||||
response.results?.forEach((result, index) => {
|
||||
memories.push(`${index + 1}. ${result.memory} \n`)
|
||||
})
|
||||
const memories = formatSearchHitsForPrompt(response.results)
|
||||
await trackEvent(eventSource)
|
||||
return { success: true, data: memories }
|
||||
} catch (error) {
|
||||
|
|
|
|||
|
|
@ -12,6 +12,8 @@
|
|||
"zip": "wxt zip",
|
||||
"zip:firefox": "wxt zip -b firefox",
|
||||
"compile": "tsc --noEmit",
|
||||
"test": "bun test",
|
||||
"eval:hybrid-recall": "bun utils/hybrid-recall.eval.ts",
|
||||
"postinstall": "wxt prepare"
|
||||
},
|
||||
"dependencies": {
|
||||
|
|
|
|||
42
apps/browser-extension/utils/hybrid-recall.eval.test.ts
Normal file
42
apps/browser-extension/utils/hybrid-recall.eval.test.ts
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
import { describe, expect, it } from "bun:test"
|
||||
import {
|
||||
assertHybridRecallPass,
|
||||
evaluateHybridRecallFixtures,
|
||||
formatSearchHitsLegacy,
|
||||
renderHybridRecallReport,
|
||||
summarizeHybridRecallEval,
|
||||
} from "./hybrid-recall.eval"
|
||||
import { formatSearchHitsForPrompt } from "./search-request"
|
||||
|
||||
describe("hybrid recall evaluation", () => {
|
||||
it("prints a before/after report and meets pass thresholds", () => {
|
||||
const summary = summarizeHybridRecallEval()
|
||||
const report = renderHybridRecallReport(summary)
|
||||
console.log(`\n${report}\n`)
|
||||
|
||||
const failures = assertHybridRecallPass(summary)
|
||||
expect(failures).toEqual([])
|
||||
})
|
||||
|
||||
it("proves the legacy gap on every fixture that has chunk-only hits", () => {
|
||||
for (const row of evaluateHybridRecallFixtures()) {
|
||||
if (row.next.chunkOnlyTotal === 0) continue
|
||||
expect(row.legacy.recoveredChunkOnly).toBe(0)
|
||||
expect(row.next.recoveredChunkOnly).toBe(row.next.chunkOnlyTotal)
|
||||
expect(row.legacy.undefinedLines).toBeGreaterThan(0)
|
||||
expect(row.next.undefinedLines).toBe(0)
|
||||
}
|
||||
})
|
||||
|
||||
it("keeps contiguous numbering after skipping empty hits", () => {
|
||||
const lines = formatSearchHitsForPrompt([
|
||||
{ memory: "A" },
|
||||
{},
|
||||
{ chunk: "B" },
|
||||
])
|
||||
expect(lines.map((line) => line.match(/^(\d+)\./)?.[1])).toEqual(["1", "2"])
|
||||
expect(
|
||||
formatSearchHitsLegacy([{ memory: "A" }, {}, { chunk: "B" }]),
|
||||
).toEqual(["1. A \n", "2. undefined \n", "3. undefined \n"])
|
||||
})
|
||||
})
|
||||
296
apps/browser-extension/utils/hybrid-recall.eval.ts
Normal file
296
apps/browser-extension/utils/hybrid-recall.eval.ts
Normal file
|
|
@ -0,0 +1,296 @@
|
|||
/**
|
||||
* Offline A/B proof for hybrid recall.
|
||||
* Legacy = memory field only. Next = memory || chunk.
|
||||
*
|
||||
* Docs: docs/hybrid-recall-eval.md
|
||||
* Run: bun run eval:hybrid-recall
|
||||
*/
|
||||
import {
|
||||
buildSearchMemoriesBody,
|
||||
formatSearchHitsForPrompt,
|
||||
type SearchHit,
|
||||
} from "./search-request"
|
||||
|
||||
/** Old background.ts behavior: only `result.memory`. */
|
||||
export function formatSearchHitsLegacy(
|
||||
results: SearchHit[] | null | undefined,
|
||||
): string[] {
|
||||
if (!results?.length) return []
|
||||
return results.map((result, index) => `${index + 1}. ${result.memory} \n`)
|
||||
}
|
||||
|
||||
export type HybridRecallFixture = {
|
||||
id: string
|
||||
description: string
|
||||
/** Gold substrings that must appear in the new prompt. */
|
||||
expectedTexts: string[]
|
||||
/** Substrings that exist only as `chunk` (legacy drops these). */
|
||||
chunkOnlyTexts: string[]
|
||||
results: SearchHit[]
|
||||
}
|
||||
|
||||
/** Fixture API payloads that mix memory facts with document chunks. */
|
||||
export const HYBRID_RECALL_FIXTURES: HybridRecallFixture[] = [
|
||||
{
|
||||
id: "mixed-memory-and-chunk",
|
||||
description: "Typical hybrid page: facts + one RAG chunk",
|
||||
expectedTexts: [
|
||||
"Prefers dark mode",
|
||||
"Uses Biome for formatting",
|
||||
"Deploy checklist: run migrations before restarting the API",
|
||||
],
|
||||
chunkOnlyTexts: [
|
||||
"Deploy checklist: run migrations before restarting the API",
|
||||
],
|
||||
results: [
|
||||
{ memory: "Prefers dark mode" },
|
||||
{ memory: "Uses Biome for formatting" },
|
||||
{
|
||||
chunk: "Deploy checklist: run migrations before restarting the API",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
id: "chunk-starvation-shape",
|
||||
description:
|
||||
"Memory-heavy hybrid response with one surviving document chunk",
|
||||
expectedTexts: [
|
||||
"User works at Acme",
|
||||
"Timezone is America/New_York",
|
||||
"Prefers TypeScript, strict mode",
|
||||
"Onboarding doc: VPN must be connected before staging deploy",
|
||||
],
|
||||
chunkOnlyTexts: [
|
||||
"Onboarding doc: VPN must be connected before staging deploy",
|
||||
],
|
||||
results: [
|
||||
{ memory: "User works at Acme" },
|
||||
{ memory: "Timezone is America/New_York" },
|
||||
{ memory: "Prefers TypeScript, strict mode" },
|
||||
{
|
||||
chunk: "Onboarding doc: VPN must be connected before staging deploy",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
id: "chunk-only-page",
|
||||
description: "Document-only hits with no extracted memories",
|
||||
expectedTexts: [
|
||||
"Incident runbook: page the on-call if error rate > 5%",
|
||||
"Rollback: redeploy previous image tag from the deploy board",
|
||||
],
|
||||
chunkOnlyTexts: [
|
||||
"Incident runbook: page the on-call if error rate > 5%",
|
||||
"Rollback: redeploy previous image tag from the deploy board",
|
||||
],
|
||||
results: [
|
||||
{ chunk: "Incident runbook: page the on-call if error rate > 5%" },
|
||||
{
|
||||
chunk: "Rollback: redeploy previous image tag from the deploy board",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
id: "empty-and-whitespace-noise",
|
||||
description:
|
||||
"Whitespace-only memory must not block a valid chunk; blanks invent no lines",
|
||||
expectedTexts: ["Keep this fact", "Important document text"],
|
||||
chunkOnlyTexts: ["Important document text"],
|
||||
results: [
|
||||
{ memory: "Keep this fact" },
|
||||
{ memory: " ", chunk: "Important document text" },
|
||||
{ chunk: "" },
|
||||
{},
|
||||
],
|
||||
},
|
||||
]
|
||||
|
||||
export type FormatterMetrics = {
|
||||
undefinedLines: number
|
||||
usableLines: number
|
||||
recoveredExpected: number
|
||||
recoveredChunkOnly: number
|
||||
expectedTotal: number
|
||||
chunkOnlyTotal: number
|
||||
}
|
||||
|
||||
export type FixtureEvalResult = {
|
||||
id: string
|
||||
description: string
|
||||
legacy: FormatterMetrics
|
||||
next: FormatterMetrics
|
||||
legacyLines: string[]
|
||||
nextLines: string[]
|
||||
}
|
||||
|
||||
function countMatches(lines: string[], needles: string[]): number {
|
||||
return needles.filter((needle) => lines.some((line) => line.includes(needle)))
|
||||
.length
|
||||
}
|
||||
|
||||
export function scoreFormatter(
|
||||
lines: string[],
|
||||
fixture: HybridRecallFixture,
|
||||
): FormatterMetrics {
|
||||
const undefinedLines = lines.filter((line) =>
|
||||
/\bundefined\b/.test(line),
|
||||
).length
|
||||
const usableLines = lines.filter(
|
||||
(line) =>
|
||||
!/\bundefined\b/.test(line) &&
|
||||
line.replace(/^\d+\.\s*/, "").trim().length > 0,
|
||||
).length
|
||||
|
||||
return {
|
||||
undefinedLines,
|
||||
usableLines,
|
||||
recoveredExpected: countMatches(lines, fixture.expectedTexts),
|
||||
recoveredChunkOnly: countMatches(lines, fixture.chunkOnlyTexts),
|
||||
expectedTotal: fixture.expectedTexts.length,
|
||||
chunkOnlyTotal: fixture.chunkOnlyTexts.length,
|
||||
}
|
||||
}
|
||||
|
||||
export function evaluateHybridRecallFixtures(
|
||||
fixtures: HybridRecallFixture[] = HYBRID_RECALL_FIXTURES,
|
||||
): FixtureEvalResult[] {
|
||||
return fixtures.map((fixture) => {
|
||||
const legacyLines = formatSearchHitsLegacy(fixture.results)
|
||||
const nextLines = formatSearchHitsForPrompt(fixture.results)
|
||||
return {
|
||||
id: fixture.id,
|
||||
description: fixture.description,
|
||||
legacy: scoreFormatter(legacyLines, fixture),
|
||||
next: scoreFormatter(nextLines, fixture),
|
||||
legacyLines,
|
||||
nextLines,
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
export type EvalSummary = {
|
||||
requestUsesHybrid: boolean
|
||||
fixtureCount: number
|
||||
legacyUndefinedLines: number
|
||||
nextUndefinedLines: number
|
||||
legacyChunkRecovery: number
|
||||
nextChunkRecovery: number
|
||||
chunkOnlyTotal: number
|
||||
legacyExpectedRecovery: number
|
||||
nextExpectedRecovery: number
|
||||
expectedTotal: number
|
||||
results: FixtureEvalResult[]
|
||||
}
|
||||
|
||||
export function summarizeHybridRecallEval(
|
||||
results: FixtureEvalResult[] = evaluateHybridRecallFixtures(),
|
||||
): EvalSummary {
|
||||
const body = buildSearchMemoriesBody("eval query")
|
||||
const sum = (pick: (row: FixtureEvalResult) => number) =>
|
||||
results.reduce((total, row) => total + pick(row), 0)
|
||||
|
||||
return {
|
||||
requestUsesHybrid: body.searchMode === "hybrid",
|
||||
fixtureCount: results.length,
|
||||
legacyUndefinedLines: sum((row) => row.legacy.undefinedLines),
|
||||
nextUndefinedLines: sum((row) => row.next.undefinedLines),
|
||||
legacyChunkRecovery: sum((row) => row.legacy.recoveredChunkOnly),
|
||||
nextChunkRecovery: sum((row) => row.next.recoveredChunkOnly),
|
||||
chunkOnlyTotal: sum((row) => row.next.chunkOnlyTotal),
|
||||
legacyExpectedRecovery: sum((row) => row.legacy.recoveredExpected),
|
||||
nextExpectedRecovery: sum((row) => row.next.recoveredExpected),
|
||||
expectedTotal: sum((row) => row.next.expectedTotal),
|
||||
results,
|
||||
}
|
||||
}
|
||||
|
||||
/** Returns failure messages; empty array means PASS. */
|
||||
export function assertHybridRecallPass(summary: EvalSummary): string[] {
|
||||
const failures: string[] = []
|
||||
|
||||
if (!summary.requestUsesHybrid) {
|
||||
failures.push("search body must set searchMode: hybrid")
|
||||
}
|
||||
if (summary.nextUndefinedLines !== 0) {
|
||||
failures.push(
|
||||
`new formatter still emits undefined lines (${summary.nextUndefinedLines})`,
|
||||
)
|
||||
}
|
||||
if (summary.legacyUndefinedLines === 0) {
|
||||
failures.push(
|
||||
"legacy formatter produced no undefined lines — fixture no longer proves the gap",
|
||||
)
|
||||
}
|
||||
if (summary.nextChunkRecovery !== summary.chunkOnlyTotal) {
|
||||
failures.push(
|
||||
`chunk recovery ${summary.nextChunkRecovery}/${summary.chunkOnlyTotal}`,
|
||||
)
|
||||
}
|
||||
if (summary.legacyChunkRecovery !== 0) {
|
||||
failures.push(
|
||||
`legacy unexpectedly recovered chunk-only text (${summary.legacyChunkRecovery})`,
|
||||
)
|
||||
}
|
||||
if (summary.nextExpectedRecovery !== summary.expectedTotal) {
|
||||
failures.push(
|
||||
`expected-text recovery ${summary.nextExpectedRecovery}/${summary.expectedTotal}`,
|
||||
)
|
||||
}
|
||||
if (summary.nextExpectedRecovery <= summary.legacyExpectedRecovery) {
|
||||
failures.push(
|
||||
`new recovery (${summary.nextExpectedRecovery}) did not beat legacy (${summary.legacyExpectedRecovery})`,
|
||||
)
|
||||
}
|
||||
|
||||
return failures
|
||||
}
|
||||
|
||||
export function renderHybridRecallReport(summary: EvalSummary): string {
|
||||
const lines: string[] = [
|
||||
"Hybrid recall evaluation",
|
||||
"========================",
|
||||
`fixtures: ${summary.fixtureCount}`,
|
||||
`request searchMode hybrid: ${summary.requestUsesHybrid}`,
|
||||
"",
|
||||
"Aggregate",
|
||||
"---------",
|
||||
`undefined lines legacy=${summary.legacyUndefinedLines} next=${summary.nextUndefinedLines}`,
|
||||
`chunk-only recall legacy=${summary.legacyChunkRecovery}/${summary.chunkOnlyTotal} next=${summary.nextChunkRecovery}/${summary.chunkOnlyTotal}`,
|
||||
`expected recall legacy=${summary.legacyExpectedRecovery}/${summary.expectedTotal} next=${summary.nextExpectedRecovery}/${summary.expectedTotal}`,
|
||||
"",
|
||||
]
|
||||
|
||||
for (const row of summary.results) {
|
||||
lines.push(`## ${row.id}`)
|
||||
lines.push(row.description)
|
||||
lines.push(
|
||||
`legacy: undef=${row.legacy.undefinedLines} usable=${row.legacy.usableLines} chunks=${row.legacy.recoveredChunkOnly}/${row.legacy.chunkOnlyTotal}`,
|
||||
)
|
||||
lines.push(
|
||||
`next: undef=${row.next.undefinedLines} usable=${row.next.usableLines} chunks=${row.next.recoveredChunkOnly}/${row.next.chunkOnlyTotal}`,
|
||||
)
|
||||
lines.push("legacy prompt:")
|
||||
for (const line of row.legacyLines) lines.push(` ${JSON.stringify(line)}`)
|
||||
lines.push("next prompt:")
|
||||
for (const line of row.nextLines) lines.push(` ${JSON.stringify(line)}`)
|
||||
lines.push("")
|
||||
}
|
||||
|
||||
const failures = assertHybridRecallPass(summary)
|
||||
if (failures.length === 0) {
|
||||
lines.push(
|
||||
"PASS: new path recovers all fixture context; legacy gap confirmed.",
|
||||
)
|
||||
} else {
|
||||
lines.push("FAIL:")
|
||||
for (const failure of failures) lines.push(` - ${failure}`)
|
||||
}
|
||||
|
||||
return lines.join("\n")
|
||||
}
|
||||
|
||||
if (import.meta.main) {
|
||||
const summary = summarizeHybridRecallEval()
|
||||
console.log(renderHybridRecallReport(summary))
|
||||
if (assertHybridRecallPass(summary).length > 0) process.exit(1)
|
||||
}
|
||||
|
|
@ -1,10 +1,15 @@
|
|||
import { describe, expect, it } from "bun:test"
|
||||
import { buildSearchMemoriesBody } from "./search-request"
|
||||
import {
|
||||
buildSearchMemoriesBody,
|
||||
formatSearchHitText,
|
||||
formatSearchHitsForPrompt,
|
||||
} from "./search-request"
|
||||
|
||||
describe("buildSearchMemoriesBody", () => {
|
||||
it("builds the default related-memory search body", () => {
|
||||
it("defaults to hybrid search with related memories", () => {
|
||||
expect(buildSearchMemoriesBody("deploy notes")).toEqual({
|
||||
q: "deploy notes",
|
||||
searchMode: "hybrid",
|
||||
include: { relatedMemories: true },
|
||||
})
|
||||
})
|
||||
|
|
@ -12,8 +17,52 @@ describe("buildSearchMemoriesBody", () => {
|
|||
it("includes the container tag when provided", () => {
|
||||
expect(buildSearchMemoriesBody("deploy notes", "sm_project_docs")).toEqual({
|
||||
q: "deploy notes",
|
||||
searchMode: "hybrid",
|
||||
include: { relatedMemories: true },
|
||||
containerTag: "sm_project_docs",
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
describe("formatSearchHitText", () => {
|
||||
it("prefers memory text over chunk text", () => {
|
||||
expect(
|
||||
formatSearchHitText({ memory: "Likes TypeScript", chunk: "doc chunk" }),
|
||||
).toBe("Likes TypeScript")
|
||||
})
|
||||
|
||||
it("falls back to chunk when memory is missing", () => {
|
||||
expect(formatSearchHitText({ chunk: " RAG chunk text " })).toBe(
|
||||
"RAG chunk text",
|
||||
)
|
||||
})
|
||||
|
||||
it("falls back to chunk when memory is only whitespace", () => {
|
||||
expect(
|
||||
formatSearchHitText({
|
||||
memory: " ",
|
||||
chunk: "Important document text",
|
||||
}),
|
||||
).toBe("Important document text")
|
||||
})
|
||||
|
||||
it("returns null for empty hits", () => {
|
||||
expect(formatSearchHitText({})).toBeNull()
|
||||
expect(formatSearchHitText({ memory: " " })).toBeNull()
|
||||
})
|
||||
})
|
||||
|
||||
describe("formatSearchHitsForPrompt", () => {
|
||||
it("numbers contiguous lines and skips empty hybrid hits", () => {
|
||||
expect(
|
||||
formatSearchHitsForPrompt([
|
||||
{ memory: "Prefers dark mode" },
|
||||
{ memory: undefined, chunk: undefined },
|
||||
{ chunk: "Shipping docs mention staging URLs" },
|
||||
]),
|
||||
).toEqual([
|
||||
"1. Prefers dark mode \n",
|
||||
"2. Shipping docs mention staging URLs \n",
|
||||
])
|
||||
})
|
||||
})
|
||||
|
|
|
|||
|
|
@ -1,14 +1,43 @@
|
|||
export type SearchHit = {
|
||||
memory?: string | null
|
||||
chunk?: string | null
|
||||
}
|
||||
|
||||
export function buildSearchMemoriesBody(
|
||||
query: string,
|
||||
containerTag?: string,
|
||||
): {
|
||||
q: string
|
||||
searchMode: "hybrid"
|
||||
include: { relatedMemories: boolean }
|
||||
containerTag?: string
|
||||
} {
|
||||
return {
|
||||
q: query,
|
||||
// API default is "memories"; hybrid also returns document chunks.
|
||||
searchMode: "hybrid",
|
||||
include: { relatedMemories: true },
|
||||
...(containerTag ? { containerTag } : {}),
|
||||
}
|
||||
}
|
||||
|
||||
/** Prefer extracted memory text; fall back to document chunk. */
|
||||
export function formatSearchHitText(hit: SearchHit): string | null {
|
||||
const memory = typeof hit.memory === "string" ? hit.memory.trim() : ""
|
||||
const chunk = typeof hit.chunk === "string" ? hit.chunk.trim() : ""
|
||||
const text = memory || chunk
|
||||
return text.length > 0 ? text : null
|
||||
}
|
||||
|
||||
/** Numbered prompt lines for Included Memories; skips empty hits. */
|
||||
export function formatSearchHitsForPrompt(
|
||||
results: SearchHit[] | null | undefined,
|
||||
): string[] {
|
||||
if (!results?.length) return []
|
||||
const lines: string[] = []
|
||||
for (const hit of results) {
|
||||
const text = formatSearchHitText(hit)
|
||||
if (text) lines.push(`${lines.length + 1}. ${text} \n`)
|
||||
}
|
||||
return lines
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue