supermemory/apps/docs/concepts/how-it-works.mdx
sohamd22 4347970d16 docs(api): align v5 ingestion and document operations docs with the API (#1810)
Fixes v5 ingestion and document-operation docs.

- **Query params return `400`.** `dreaming`, `fileType` and `mimeType` were documented as query params; they are JSON body or multipart form fields.
- **Errors.** Out-of-balance quota is `402`, not `429`; `429` is only rate limiting. Added the `409` returned during a namespace merge.
- **Statuses.** There is no `processing` status. The real ones are queued, extracting, chunking, embedding, done and failed; treat anything else as in progress.
- **`group` on updates.** `group` is ingestion-side: it picks which existing memories serve as context while content is processed. Changing it only takes effect on re-ingestion, so it's sent with `content`/`file`. The "group-only PATCH" claims are removed from the migration doc.
- **Lists, deletes and forget:**
  - Valid `sort` values per type, a `limit` max of 100, and `include.forgotten`.
  - Deletes skip documents that are still processing.
  - Forget-matching has a 100-match cap and a 2000-character `query` limit.
- **Failed documents stay `failed`.** The docs said they're deleted after 2 minutes. They actually stay `failed`, and resending the same `id` reprocesses them.
- **Smaller fixes:**
  - Metadata allows `string[]`.
  - The batch response field is `count`.
  - `audio` is a valid `fileType`.
  - Chunk overlap is 200 characters.
  - URL extraction no longer claims Markdowner.
  - Search examples no longer index the response object as an array.

Checked against the current API and the published SDKs (TS 5.0.1, Python 5.0.0). Legacy v3/v4 docs are unchanged apart from factual fixes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-10-15 18:52:40 +00:00

181 lines
7.9 KiB
Text

---
title: "How Supermemory works"
sidebarTitle: "How it works"
description: "How a file or chat turn becomes something you can search: the ingest pipeline, statuses and outputs."
icon: "/icons/hugeicons/cpu.svg"
---
At its core, supermemory is powered by a custom learning model and a graph database that we built internally.
<CardGroup cols={2}>
<Card title="Learning model">
Decides what and how to learn, what is important, when to forget, creating relations, etc.
</Card>
<Card title="Temporal vector-graph engine">
Where the learnings are actually stored, optimized for search. Fact-based temporal graph that has Vector, FTS, and graph built in.
</Card>
</CardGroup>
But, you don't have to think about the above. The interface for users is as simple as it gets.
## Get started in under a minute
<CardGroup cols={2}>
<Card title="1. Get an API key" icon="/icons/hugeicons/key-01.svg" href="https://console.supermemory.ai">
From the [developer console](https://console.supermemory.ai) — **API Keys → Create API Key**. `console.supermemory.ai` is where keys and usage live.
</Card>
<Card title="2. Use it" icon="/icons/hugeicons/command-line.svg" href="/using-supermemory">
Install the SDK, drop in your key, add a memory, and search it — right below, or the full [ingest → retrieve loop](/using-supermemory).
</Card>
</CardGroup>
<CodeGroup>
```typescript TypeScript
import { Supermemory } from "supermemory";
const supermemory = new Supermemory({ apiKey: "sm_..." }); // from console.supermemory.ai → API Keys
await supermemory.add("user_123", {
content: "The user loves Paris.",
dreaming: "instant", // memories available right after processing
});
const { results } = await supermemory.search("user_123", {
query: "where does the user want to travel?",
});
```
```python Python
# pip install supermemory
from supermemory import Supermemory
client = Supermemory(api_key="sm_...") # from console.supermemory.ai → API Keys
client.add(
"user_123",
content="The user loves Paris.",
dreaming="instant", # memories available right after processing
)
results = client.search("user_123", query="where does the user want to travel?")
```
```bash curl
curl -X POST "https://api.supermemory.ai/ns/user_123/document" \
-H "Authorization: Bearer sm_..." \
-H "Content-Type: application/json" \
-d '{ "content": "The user loves Paris.", "dreaming": "instant" }'
curl -X POST "https://api.supermemory.ai/ns/user_123/search" \
-H "Authorization: Bearer sm_..." \
-H "Content-Type: application/json" \
-d '{ "query": "where does the user want to travel?" }'
```
</CodeGroup>
## What you send: documents
A **document** is raw input — whatever you hand Supermemory:
- Conversation transcripts and messages
- Text, markdown, HTML
- PDFs, images, audio/video, code
- URLs and connector items (Drive, Notion, Gmail, …)
You do not pre-chunk or pick an embedding model. See [Multi-modal ingestion](/concepts/content-types) for formats, and [Add context](/ingestion/add-memories) for the API.
Supermemory handles the ingestion and extraction for you. This also gives us a big advantage for quality - The engine extracts it in an optimized way with Contextual Chunking and other features for better quality search and memory generation.
> Use a stable **`id`** when the same conversation or file will be updated later (sessions, connector syncs). That identity also drives [diff billing](/overview/billing#full-discount-on-already-seen-tokens-diff-billing) on re-ingest.
## What the pipeline does
| Stage | What happens |
| --- | --- |
| **Queued** | Accepted; waiting to run |
| **Extracting** | Text / OCR / transcription / page fetch |
| **Chunking** | Splits content for retrieval (type-aware where needed) |
| **Embedding** | Vectors for similarity search |
| **Indexing** | Makes chunks and derived structure searchable |
| **Done** | Document path is ready for search |
```typescript
const doc = await supermemory.add("user_123", {
content: conversationText,
id: "chat_session_1",
});
// Poll until ready
const { system } = await supermemory.documents.get("user_123", doc.id);
// system.status → "queued" | "extracting" | ... | "done" | "failed"
```
Larger PDFs and long video take longer. Short chat turns usually finish in seconds.
## Dreaming (how memories enter the graph)
A document with status `done` has its chunks indexed for search. Memories (the graph's facts, updates and derived facts) come from a second phase called **dreaming**.
This is when the content is passed through the memory model and merged, arranged and organized for the future.
Pass `dreaming` on [add](/ingestion/add-memories) (a field in the JSON body, next to `content` in the SDK):
| Mode | Default? | Behavior | When to use |
| --- | --- | --- | --- |
| **`dynamic`** | Yes | Related documents are grouped so memories form from **coherent units**, not one isolated write at a time. Graph quality is higher for real multi-turn / multi-doc flows. Memory extraction may continue **after** `status: "done"`. | Production agents, connectors, ongoing sessions |
| **`instant`** | No | This document is dreamed on its own, right away. Memories are available as soon as processing finishes for that document. Bills one extra [operation](/overview/billing) per document. | Demos, quickstarts, “I need the graph now” |
```typescript
// Production default — omit or set explicitly
await supermemory.add("user_123", {
content: conversationText,
id: "chat_session_1",
dreaming: "dynamic",
});
// Need memories immediately (e.g. tutorial)
await supermemory.add("user_123", {
content: conversationText,
id: "chat_session_1",
dreaming: "instant",
});
```
**Rule of thumb:** use `dynamic` in real apps for better quality and cost, because batching lets new memories connect to related ones. Use `instant` when the very next step is a memory search or profile that must already reflect this document, as in the [quickstart](/quickstart).
How those memories connect and stay true over time is [Graph memory](/concepts/graph-memory). API detail: [Processing modes](/ingestion/add-memories#processing-modes).
## What you get out
After the pipeline runs, the same document leads to three things -> Chunks, Memories and Profile. (in the same `namespace`):
| Output | Role | Go deeper |
| --- | --- | --- |
| **Document chunks** | Grounding in the raw source (RAG / SuperRAG) | [SuperRAG](/concepts/super-rag), [Search API](/recall/search) |
| **Memories** | Extracted facts in a living graph — updates, links, time | [Graph memory](/concepts/graph-memory) |
| **Profile** | A sample of memories, static + dynamic summary for always-on context | [Profiles](/concepts/user-profiles), [Profile API](/recall/user-profiles) |
Supermemory does more than store the file. It derives memories (what it understood) and keeps chunks (the source), so you can both personalize and ground answers. That distinction is the core of [Memory vs RAG](/concepts/memory-vs-rag).
## Isolation and identity
- **`namespace`** — hard isolation boundary (user, tenant, project), carried in the URL. See [Namespaces](/concepts/container-tags).
- **Metadata** — soft dimensions *inside* a namespace for filtering. See [Metadata filtering](/concepts/filtering).
- **Scoped API keys** — credentials that cannot cross a namespace. See [API keys](/authentication#scoped-api-keys).
## Next steps
<CardGroup cols={2}>
<Card title="Graph memory" icon="/icons/hugeicons/share-08.svg" href="/concepts/graph-memory">
How facts connect, update, and stay true over time.
</Card>
<Card title="Multi-modal ingestion" icon="/icons/hugeicons/files-01.svg" href="/concepts/content-types">
Formats, extractors, and what you can send.
</Card>
<Card title="Add context" icon="/icons/hugeicons/plus-sign.svg" href="/ingestion/add-memories">
API: add, id, files, dreaming, status.
</Card>
<Card title="Search API" icon="/icons/hugeicons/search-01.svg" href="/recall/search">
Query documents and memories after the pipeline finishes.
</Card>
</CardGroup>