feat(extension): use hybrid search and surface document chunks in recall

Auto-recall called /v4/search with the memories-only default and formatted
hits as result.memory, so document chunk results became prompt lines with
the literal string undefined. Request searchMode hybrid and format with
memory || chunk. Includes an offline A/B eval harness and proof doc.
This commit is contained in:
James Yang 2026-08-06 15:33:54 -04:00
parent 069b8c373c
commit 4d02f6d79c
8 changed files with 599 additions and 8 deletions

View file

@ -1 +1,9 @@
## supermemory Browser Extension
## supermemory Browser Extension
### Hybrid recall evaluation
Auto-recall proof (legacy `memory`-only formatting vs hybrid `memory || chunk`):
- **Doc:** [docs/hybrid-recall-eval.md](./docs/hybrid-recall-eval.md)
- **Report:** `bun run eval:hybrid-recall`
- **Tests:** `bun test utils/hybrid-recall.eval.test.ts`

View file

@ -0,0 +1,177 @@
# Hybrid recall — evaluation & proof
This doc explains **what we claim**, **how we prove it**, and **how to re-run the proof** for the browser-extension auto-recall change on branch `feat/extension-hybrid-recall`.
## Claim (precise)
Auto-recall in the extension used to:
1. Call `/v4/search` **without** `searchMode` (API default = `"memories"`), and
2. Format each hit as `` `${result.memory}` `` only.
When the API returns a **document chunk** (`chunk` set, `memory` absent), the injected “Included Memories” prompt got the literal string `undefined` and **lost** that document context.
**After this change**, auto-recall:
1. Requests `searchMode: "hybrid"` (memories **and** document chunks), and
2. Formats each hit as `memory || chunk`, skipping empty hits and renumbering contiguously.
We claim an improvement in **context the extension injects into the LLM**, not an improvement to Supermemory’s private ranking/embedding engine.
| In scope | Out of scope |
|----------|----------------|
| Request body uses hybrid | Server ranking / fusion quality |
| Prompt lines no longer contain `undefined` | MemoryBench provider scores |
| Chunk-only API hits become usable prompt text | Changing how memories are stored |
| Empty hits don’t invent numbered lines | End-to-end browser chrome E2E |
## How the proof works
We run an **offline A/B** on fixed gold fixtures that look like hybrid `/v4/search` payloads:
```
same fixture results
│
├─► legacy formatter → metrics L
│ (memory field only — old background.ts)
│
└─► new formatter → metrics N
(memory || chunk — search-request.ts)
```
Fixtures live in `utils/hybrid-recall.eval.ts` (`HYBRID_RECALL_FIXTURES`):
| Fixture id | What it models |
|------------|----------------|
| `mixed-memory-and-chunk` | Typical mix: extracted facts + one RAG chunk |
| `chunk-starvation-shape` | Memory-heavy list with a single surviving doc chunk |
| `chunk-only-page` | Document-only hits (no extracted memories) |
| `empty-and-whitespace-noise` | Garbage / blank hits must not pollute the prompt |
### Metrics
| Metric | Meaning |
|--------|---------|
| **undefined lines** | Prompt lines matching `\bundefined\b` (legacy bug signature) |
| **chunk-only recall** | How many fixture strings that exist *only* as `chunk` appear in the prompt |
| **expected recall** | How many gold substrings appear in the prompt |
| **usable lines** | Non-empty lines that are not `undefined` |
| **requestUsesHybrid** | `buildSearchMemoriesBody` sets `searchMode: "hybrid"` |
### Pass gates (`assertHybridRecallPass`)
The harness **fails CI-style** unless all of these hold:
1. Search body uses `searchMode: "hybrid"`.
2. New formatter: **0** undefined lines.
3. Legacy formatter: **> 0** undefined lines (fixtures still prove the gap).
4. New recovers **all** chunk-only gold strings; legacy recovers **none**.
5. New recovers **all** expected strings and **beats** legacy expected recovery.
That combination both **shows the bug** and **shows the fix**.
## Run the proof
From `apps/browser-extension`:
```bash
# Human-readable before/after report (exit 1 on FAIL)
bun run eval:hybrid-recall
# Same gates as automated tests (+ unit coverage)
bun test utils/hybrid-recall.eval.test.ts utils/search-request.test.ts
```
Equivalent:
```bash
bun utils/hybrid-recall.eval.ts
bun test
```
## Latest local result (re-run to refresh)
Captured with `bun run eval:hybrid-recall`:
```
fixtures: 4
request searchMode hybrid: true
undefined lines legacy=6 next=0
chunk-only recall legacy=0/4 next=4/4
expected recall legacy=6/10 next=10/10
PASS: new path recovers all fixture context; legacy gap confirmed.
```
### Example before / after (one fixture)
**Fixture:** `mixed-memory-and-chunk`
| | Legacy | Next |
|---|--------|------|
| Line 1 | Prefers dark mode | Prefers dark mode |
| Line 2 | Uses Biome for formatting | Uses Biome for formatting |
| Line 3 | **`undefined`** | Deploy checklist: run migrations before restarting the API |
The third hit is chunk-only. Legacy injects noise; next injects the document text the model can use.
### Chunk-only page (strongest gap)
**Fixture:** `chunk-only-page`
| Legacy | Next |
|--------|------|
| `1. undefined` | Incident runbook… |
| `2. undefined` | Rollback: redeploy… |
Legacy usable context: **0**. Next: **2/2** document lines.
## How to present this (PR / review)
Suggested PR section:
```markdown
## Proof
Offline A/B eval: `bun run eval:hybrid-recall` in `apps/browser-extension`.
| Metric | Legacy | Next |
|--------|--------|------|
| undefined prompt lines | 6 | 0 |
| chunk-only texts recovered | 0/4 | 4/4 |
| expected texts recovered | 6/10 | 10/10 |
Doc: `apps/browser-extension/docs/hybrid-recall-eval.md`
Harness: `utils/hybrid-recall.eval.ts`
```
Attach the full CLI report as a PR comment or screenshot of the test output.
## Manual smoke (optional, not automated)
Use when you want confidence against a **live** API (requires your account + loaded extension build):
1. Load the extension from a local `bun run build` / `bun run dev` output.
2. In one Supermemory project, save (a) a short preference fact and (b) a multi-sentence note/doc whose wording won’t be extracted as a memory yet (or search in a way that returns chunks).
3. On ChatGPT/Claude/T3, type a query that should hit both.
4. Open **Included Memories**.
5. Confirm: no `undefined` rows; document wording appears when the hit is a chunk.
This smoke check is complementary. The offline harness is the **regression-proof** gate; smoke validates wiring to a real `/v4/search`.
## What this does *not* replace
- **MemoryBench** (`supermemoryai/memorybench`) scores providers’ ingest→search→answer quality. It does not exercise this Chrome content-script formatter.
- Server-side hybrid ranking quirks (e.g. memory-heavy containers starving chunks — see supermemory#1398) are unchanged by this PR. We only ensure that **when** a chunk is returned, the extension uses it.
## File map
| Path | Role |
|------|------|
| `utils/search-request.ts` | `searchMode: "hybrid"` + `formatSearchHitsForPrompt` |
| `entrypoints/background.ts` | Wires formatter into `GET_RELATED_MEMORIES` |
| `utils/hybrid-recall.eval.ts` | Fixtures, legacy vs next scoring, CLI report |
| `utils/hybrid-recall.eval.test.ts` | Hard pass gates in `bun test` |
| `utils/search-request.test.ts` | Unit tests for body + hit formatting |
| `docs/hybrid-recall-eval.md` | This document |

View file

@ -9,6 +9,7 @@ import {
MESSAGE_TYPES,
POSTHOG_EVENT_KEY,
} from "../utils/constants"
import { formatSearchHitsForPrompt } from "../utils/search-request"
import { trackEvent } from "../utils/posthog"
import { captureTwitterTokens } from "../utils/twitter-auth"
import {
@ -226,12 +227,9 @@ export default defineBackground(() => {
const responseData = await searchMemories(data, containerTag)
const response = responseData as {
results?: Array<{ memory?: string }>
results?: Array<{ memory?: string; chunk?: string }>
}
const memories: string[] = []
response.results?.forEach((result, index) => {
memories.push(`${index + 1}. ${result.memory} \n`)
})
const memories = formatSearchHitsForPrompt(response.results)
await trackEvent(eventSource)
return { success: true, data: memories }
} catch (error) {

View file

@ -12,6 +12,8 @@
"zip": "wxt zip",
"zip:firefox": "wxt zip -b firefox",
"compile": "tsc --noEmit",
"test": "bun test",
"eval:hybrid-recall": "bun utils/hybrid-recall.eval.ts",
"postinstall": "wxt prepare"
},
"dependencies": {

View file

@ -0,0 +1,42 @@
import { describe, expect, it } from "bun:test"
import {
assertHybridRecallPass,
evaluateHybridRecallFixtures,
formatSearchHitsLegacy,
renderHybridRecallReport,
summarizeHybridRecallEval,
} from "./hybrid-recall.eval"
import { formatSearchHitsForPrompt } from "./search-request"
describe("hybrid recall evaluation", () => {
it("prints a before/after report and meets pass thresholds", () => {
const summary = summarizeHybridRecallEval()
const report = renderHybridRecallReport(summary)
console.log(`\n${report}\n`)
const failures = assertHybridRecallPass(summary)
expect(failures).toEqual([])
})
it("proves the legacy gap on every fixture that has chunk-only hits", () => {
for (const row of evaluateHybridRecallFixtures()) {
if (row.next.chunkOnlyTotal === 0) continue
expect(row.legacy.recoveredChunkOnly).toBe(0)
expect(row.next.recoveredChunkOnly).toBe(row.next.chunkOnlyTotal)
expect(row.legacy.undefinedLines).toBeGreaterThan(0)
expect(row.next.undefinedLines).toBe(0)
}
})
it("keeps contiguous numbering after skipping empty hits", () => {
const lines = formatSearchHitsForPrompt([
{ memory: "A" },
{},
{ chunk: "B" },
])
expect(lines.map((line) => line.match(/^(\d+)\./)?.[1])).toEqual(["1", "2"])
expect(
formatSearchHitsLegacy([{ memory: "A" }, {}, { chunk: "B" }]),
).toEqual(["1. A \n", "2. undefined \n", "3. undefined \n"])
})
})

View file

@ -0,0 +1,295 @@
/**
* Offline A/B proof for hybrid recall.
* Legacy = memory field only. Next = memory || chunk.
*
* Docs: docs/hybrid-recall-eval.md
* Run: bun run eval:hybrid-recall
*/
import {
buildSearchMemoriesBody,
formatSearchHitsForPrompt,
type SearchHit,
} from "./search-request"
/** Old background.ts behavior: only `result.memory`. */
export function formatSearchHitsLegacy(
results: SearchHit[] | null | undefined,
): string[] {
if (!results?.length) return []
return results.map((result, index) => `${index + 1}. ${result.memory} \n`)
}
export type HybridRecallFixture = {
id: string
description: string
/** Gold substrings that must appear in the new prompt. */
expectedTexts: string[]
/** Substrings that exist only as `chunk` (legacy drops these). */
chunkOnlyTexts: string[]
results: SearchHit[]
}
/** Fixture API payloads that mix memory facts with document chunks. */
export const HYBRID_RECALL_FIXTURES: HybridRecallFixture[] = [
{
id: "mixed-memory-and-chunk",
description: "Typical hybrid page: facts + one RAG chunk",
expectedTexts: [
"Prefers dark mode",
"Uses Biome for formatting",
"Deploy checklist: run migrations before restarting the API",
],
chunkOnlyTexts: [
"Deploy checklist: run migrations before restarting the API",
],
results: [
{ memory: "Prefers dark mode" },
{ memory: "Uses Biome for formatting" },
{
chunk: "Deploy checklist: run migrations before restarting the API",
},
],
},
{
id: "chunk-starvation-shape",
description:
"Memory-heavy hybrid response with one surviving document chunk",
expectedTexts: [
"User works at Acme",
"Timezone is America/New_York",
"Prefers TypeScript, strict mode",
"Onboarding doc: VPN must be connected before staging deploy",
],
chunkOnlyTexts: [
"Onboarding doc: VPN must be connected before staging deploy",
],
results: [
{ memory: "User works at Acme" },
{ memory: "Timezone is America/New_York" },
{ memory: "Prefers TypeScript, strict mode" },
{
chunk: "Onboarding doc: VPN must be connected before staging deploy",
},
],
},
{
id: "chunk-only-page",
description: "Document-only hits with no extracted memories",
expectedTexts: [
"Incident runbook: page the on-call if error rate > 5%",
"Rollback: redeploy previous image tag from the deploy board",
],
chunkOnlyTexts: [
"Incident runbook: page the on-call if error rate > 5%",
"Rollback: redeploy previous image tag from the deploy board",
],
results: [
{ chunk: "Incident runbook: page the on-call if error rate > 5%" },
{
chunk: "Rollback: redeploy previous image tag from the deploy board",
},
],
},
{
id: "empty-and-whitespace-noise",
description: "Empty / whitespace hits must not invent prompt lines",
expectedTexts: ["Keep this fact"],
chunkOnlyTexts: [],
results: [
{ memory: "Keep this fact" },
{ memory: " " },
{ chunk: "" },
{},
],
},
]
export type FormatterMetrics = {
undefinedLines: number
usableLines: number
recoveredExpected: number
recoveredChunkOnly: number
expectedTotal: number
chunkOnlyTotal: number
}
export type FixtureEvalResult = {
id: string
description: string
legacy: FormatterMetrics
next: FormatterMetrics
legacyLines: string[]
nextLines: string[]
}
function countMatches(lines: string[], needles: string[]): number {
return needles.filter((needle) => lines.some((line) => line.includes(needle)))
.length
}
export function scoreFormatter(
lines: string[],
fixture: HybridRecallFixture,
): FormatterMetrics {
const undefinedLines = lines.filter((line) =>
/\bundefined\b/.test(line),
).length
const usableLines = lines.filter(
(line) =>
!/\bundefined\b/.test(line) &&
line.replace(/^\d+\.\s*/, "").trim().length > 0,
).length
return {
undefinedLines,
usableLines,
recoveredExpected: countMatches(lines, fixture.expectedTexts),
recoveredChunkOnly: countMatches(lines, fixture.chunkOnlyTexts),
expectedTotal: fixture.expectedTexts.length,
chunkOnlyTotal: fixture.chunkOnlyTexts.length,
}
}
export function evaluateHybridRecallFixtures(
fixtures: HybridRecallFixture[] = HYBRID_RECALL_FIXTURES,
): FixtureEvalResult[] {
return fixtures.map((fixture) => {
const legacyLines = formatSearchHitsLegacy(fixture.results)
const nextLines = formatSearchHitsForPrompt(fixture.results)
return {
id: fixture.id,
description: fixture.description,
legacy: scoreFormatter(legacyLines, fixture),
next: scoreFormatter(nextLines, fixture),
legacyLines,
nextLines,
}
})
}
export type EvalSummary = {
requestUsesHybrid: boolean
fixtureCount: number
legacyUndefinedLines: number
nextUndefinedLines: number
legacyChunkRecovery: number
nextChunkRecovery: number
chunkOnlyTotal: number
legacyExpectedRecovery: number
nextExpectedRecovery: number
expectedTotal: number
results: FixtureEvalResult[]
}
export function summarizeHybridRecallEval(
results: FixtureEvalResult[] = evaluateHybridRecallFixtures(),
): EvalSummary {
const body = buildSearchMemoriesBody("eval query")
const sum = (pick: (row: FixtureEvalResult) => number) =>
results.reduce((total, row) => total + pick(row), 0)
return {
requestUsesHybrid: body.searchMode === "hybrid",
fixtureCount: results.length,
legacyUndefinedLines: sum((row) => row.legacy.undefinedLines),
nextUndefinedLines: sum((row) => row.next.undefinedLines),
legacyChunkRecovery: sum((row) => row.legacy.recoveredChunkOnly),
nextChunkRecovery: sum((row) => row.next.recoveredChunkOnly),
chunkOnlyTotal: sum((row) => row.next.chunkOnlyTotal),
legacyExpectedRecovery: sum((row) => row.legacy.recoveredExpected),
nextExpectedRecovery: sum((row) => row.next.recoveredExpected),
expectedTotal: sum((row) => row.next.expectedTotal),
results,
}
}
/** Returns failure messages; empty array means PASS. */
export function assertHybridRecallPass(summary: EvalSummary): string[] {
const failures: string[] = []
if (!summary.requestUsesHybrid) {
failures.push("search body must set searchMode: hybrid")
}
if (summary.nextUndefinedLines !== 0) {
failures.push(
`new formatter still emits undefined lines (${summary.nextUndefinedLines})`,
)
}
if (summary.legacyUndefinedLines === 0) {
failures.push(
"legacy formatter produced no undefined lines — fixture no longer proves the gap",
)
}
if (summary.nextChunkRecovery !== summary.chunkOnlyTotal) {
failures.push(
`chunk recovery ${summary.nextChunkRecovery}/${summary.chunkOnlyTotal}`,
)
}
if (summary.legacyChunkRecovery !== 0) {
failures.push(
`legacy unexpectedly recovered chunk-only text (${summary.legacyChunkRecovery})`,
)
}
if (summary.nextExpectedRecovery !== summary.expectedTotal) {
failures.push(
`expected-text recovery ${summary.nextExpectedRecovery}/${summary.expectedTotal}`,
)
}
if (summary.nextExpectedRecovery <= summary.legacyExpectedRecovery) {
failures.push(
`new recovery (${summary.nextExpectedRecovery}) did not beat legacy (${summary.legacyExpectedRecovery})`,
)
}
return failures
}
export function renderHybridRecallReport(summary: EvalSummary): string {
const lines: string[] = [
"Hybrid recall evaluation",
"========================",
`fixtures: ${summary.fixtureCount}`,
`request searchMode hybrid: ${summary.requestUsesHybrid}`,
"",
"Aggregate",
"---------",
`undefined lines legacy=${summary.legacyUndefinedLines} next=${summary.nextUndefinedLines}`,
`chunk-only recall legacy=${summary.legacyChunkRecovery}/${summary.chunkOnlyTotal} next=${summary.nextChunkRecovery}/${summary.chunkOnlyTotal}`,
`expected recall legacy=${summary.legacyExpectedRecovery}/${summary.expectedTotal} next=${summary.nextExpectedRecovery}/${summary.expectedTotal}`,
"",
]
for (const row of summary.results) {
lines.push(`## ${row.id}`)
lines.push(row.description)
lines.push(
`legacy: undef=${row.legacy.undefinedLines} usable=${row.legacy.usableLines} chunks=${row.legacy.recoveredChunkOnly}/${row.legacy.chunkOnlyTotal}`,
)
lines.push(
`next: undef=${row.next.undefinedLines} usable=${row.next.usableLines} chunks=${row.next.recoveredChunkOnly}/${row.next.chunkOnlyTotal}`,
)
lines.push("legacy prompt:")
for (const line of row.legacyLines) lines.push(` ${JSON.stringify(line)}`)
lines.push("next prompt:")
for (const line of row.nextLines) lines.push(` ${JSON.stringify(line)}`)
lines.push("")
}
const failures = assertHybridRecallPass(summary)
if (failures.length === 0) {
lines.push(
"PASS: new path recovers all fixture context; legacy gap confirmed.",
)
} else {
lines.push("FAIL:")
for (const failure of failures) lines.push(` - ${failure}`)
}
return lines.join("\n")
}
if (import.meta.main) {
const summary = summarizeHybridRecallEval()
console.log(renderHybridRecallReport(summary))
if (assertHybridRecallPass(summary).length > 0) process.exit(1)
}

View file

@ -1,10 +1,15 @@
import { describe, expect, it } from "bun:test"
import { buildSearchMemoriesBody } from "./search-request"
import {
buildSearchMemoriesBody,
formatSearchHitText,
formatSearchHitsForPrompt,
} from "./search-request"
describe("buildSearchMemoriesBody", () => {
it("builds the default related-memory search body", () => {
it("defaults to hybrid search with related memories", () => {
expect(buildSearchMemoriesBody("deploy notes")).toEqual({
q: "deploy notes",
searchMode: "hybrid",
include: { relatedMemories: true },
})
})
@ -12,8 +17,43 @@ describe("buildSearchMemoriesBody", () => {
it("includes the container tag when provided", () => {
expect(buildSearchMemoriesBody("deploy notes", "sm_project_docs")).toEqual({
q: "deploy notes",
searchMode: "hybrid",
include: { relatedMemories: true },
containerTag: "sm_project_docs",
})
})
})
describe("formatSearchHitText", () => {
it("prefers memory text over chunk text", () => {
expect(
formatSearchHitText({ memory: "Likes TypeScript", chunk: "doc chunk" }),
).toBe("Likes TypeScript")
})
it("falls back to chunk when memory is missing", () => {
expect(formatSearchHitText({ chunk: " RAG chunk text " })).toBe(
"RAG chunk text",
)
})
it("returns null for empty hits", () => {
expect(formatSearchHitText({})).toBeNull()
expect(formatSearchHitText({ memory: " " })).toBeNull()
})
})
describe("formatSearchHitsForPrompt", () => {
it("numbers contiguous lines and skips empty hybrid hits", () => {
expect(
formatSearchHitsForPrompt([
{ memory: "Prefers dark mode" },
{ memory: undefined, chunk: undefined },
{ chunk: "Shipping docs mention staging URLs" },
]),
).toEqual([
"1. Prefers dark mode \n",
"2. Shipping docs mention staging URLs \n",
])
})
})

View file

@ -1,14 +1,43 @@
export type SearchHit = {
memory?: string | null
chunk?: string | null
}
export function buildSearchMemoriesBody(
query: string,
containerTag?: string,
): {
q: string
searchMode: "hybrid"
include: { relatedMemories: boolean }
containerTag?: string
} {
return {
q: query,
// API default is "memories"; hybrid also returns document chunks.
searchMode: "hybrid",
include: { relatedMemories: true },
...(containerTag ? { containerTag } : {}),
}
}
/** Prefer extracted memory text; fall back to document chunk. */
export function formatSearchHitText(hit: SearchHit): string | null {
const text = hit.memory || hit.chunk
if (typeof text !== "string") return null
const trimmed = text.trim()
return trimmed.length > 0 ? trimmed : null
}
/** Numbered prompt lines for Included Memories; skips empty hits. */
export function formatSearchHitsForPrompt(
results: SearchHit[] | null | undefined,
): string[] {
if (!results?.length) return []
const lines: string[] = []
for (const hit of results) {
const text = formatSearchHitText(hit)
if (text) lines.push(`${lines.length + 1}. ${text} \n`)
}
return lines
}