diff --git a/apps/docs/ingestion/batch-ingest-historical-data.mdx b/apps/docs/ingestion/batch-ingest-historical-data.mdx index 534ab6c6..7bea599f 100644 --- a/apps/docs/ingestion/batch-ingest-historical-data.mdx +++ b/apps/docs/ingestion/batch-ingest-historical-data.mdx @@ -108,7 +108,7 @@ Batch results preserve input order. Inspect `success`, `failed`, and every item **Endpoint:** [`GET /ns/{namespace}/document/{id}`](/api-reference/documents/get-document) -The batch endpoint returns after accepting the documents. If a later step depends on the documents being indexed, poll the returned document IDs until `system.status` is `done`. If that later step also needs the extracted memories, send the batch with `dreaming: "instant"`; under the default `dynamic` mode memory extraction is batched and can lag `done` by minutes. +The batch endpoint returns after accepting the documents. If a later step depends on the documents being indexed, poll the returned document IDs until `system.status` is `done`. If that later step also needs the extracted memories, send the batch with `dreaming: "instant"`; under the default `dynamic` mode memory extraction is batched and can lag `done` by minutes. Each document reports `system.dreamingStatus: "done"` once its memories are ready. diff --git a/apps/docs/ingestion/document-operations.mdx b/apps/docs/ingestion/document-operations.mdx index 5c648f57..8234b900 100644 --- a/apps/docs/ingestion/document-operations.mdx +++ b/apps/docs/ingestion/document-operations.mdx @@ -52,6 +52,7 @@ Retrieve paginated documents with filtering. One `list` call serves documents, c "metadata": { "source": "slack" }, "system": { "status": "done", + "dreamingStatus": "done", "createdAt": "2024-01-15T10:30:00Z", "updatedAt": "2024-01-15T10:30:00Z" } @@ -133,6 +134,7 @@ Get a specific document with its processing status. Pass `include` to include it }); console.log(doc.system.status); // "queued" | "processing" | "done" | "failed" + console.log(doc.system.dreamingStatus); // "dreaming" | "done" console.log(doc.content); ``` @@ -145,6 +147,7 @@ Get a specific document with its processing status. Pass `include` to include it ) print(doc.system.status) # "queued" | "processing" | "done" | "failed" + print(doc.system.dreaming_status) # "dreaming" | "done" print(doc.content) ``` @@ -169,6 +172,17 @@ The `id` may be the Supermemory document ID or the `id` you supplied at ingestio | `done` | Ready for search | | `failed` | Processing failed | +### Memory generation status + +`system.dreamingStatus` tracks memory generation separately from `system.status`. + +| Status | Description | +|--------|-------------| +| `dreaming` | Memories for the latest content are queued, being generated, or waiting for a retry | +| `done` | Memories are generated, or the document needed none | + +With the default `dreaming: "dynamic"`, Supermemory batches related documents before forming memories, so `status` can be `done` minutes before `dreamingStatus` is. If your next step reads memories or profiles, wait for `dreamingStatus` to be `done` too. + ```typescript async function waitForProcessing(namespace: string, id: string) { @@ -287,7 +301,7 @@ Deletes are permanent — no recovery. ## Processing queue -There is no separate processing endpoint. List documents and read `system.status` on each item. +There is no separate processing endpoint. List documents and read `system.status` and `system.dreamingStatus` on each item. @@ -296,14 +310,16 @@ There is no separate processing endpoint. List documents and read `system.status limit: 100, }); const processing = documents.filter(d => d.system.status !== "done" && d.system.status !== "failed"); - console.log(`${processing.length} documents processing`); + const dreaming = documents.filter(d => d.system.dreamingStatus === "dreaming"); + console.log(`${processing.length} documents processing, ${dreaming.length} forming memories`); ``` ```python page = client.list("user_123", "documents", limit=100) processing = [d for d in page.documents if d.system.status not in ("done", "failed")] - print(f"{len(processing)} documents processing") + dreaming = [d for d in page.documents if d.system.dreaming_status == "dreaming"] + print(f"{len(processing)} documents processing, {len(dreaming)} forming memories") ``` diff --git a/apps/docs/quickstart.mdx b/apps/docs/quickstart.mdx index d3943eac..9d528c3a 100644 --- a/apps/docs/quickstart.mdx +++ b/apps/docs/quickstart.mdx @@ -219,7 +219,7 @@ curl "https://api.supermemory.ai/ns/user_4f8a/document/DOC_ID" \ ``` -Short text with instant dreaming usually finishes in a few seconds. Larger PDFs take longer. If you omit `dreaming` (default `"dynamic"`), document RAG can work after `done` while memory extraction may still be batching — use `"instant"` when the next step is memory search or profiles. +Short text with instant dreaming usually finishes in a few seconds. Larger PDFs take longer. If you omit `dreaming` (default `"dynamic"`), document RAG can work after `done` while memory extraction may still be batching — use `"instant"` when the next step is memory search or profiles. `system.dreamingStatus` turns `done` once a document's memories are ready. ## 4. Three ways to get context back diff --git a/apps/docs/snippets/api-v5-content-management.mdx b/apps/docs/snippets/api-v5-content-management.mdx index 86bc253d..ecbf6dc0 100644 --- a/apps/docs/snippets/api-v5-content-management.mdx +++ b/apps/docs/snippets/api-v5-content-management.mdx @@ -16,11 +16,13 @@ GET /ns/{namespace}/document/{id}?include=chunks,memories Pass `include` as a comma-separated list to return chunks, memories, or both. Keys you omit are absent; requested keys with no results are empty arrays. Lifecycle fields move under `system`: ```json -{"system":{"status":"done","createdAt":"...","updatedAt":"..."}} +{"system":{"status":"done","dreamingStatus":"dreaming","createdAt":"...","updatedAt":"..."}} ``` `system.status` is one of `unknown`, `queued`, `extracting`, `chunking`, `embedding`, `indexing`, `done`, or `failed`. +`system.dreamingStatus` tracks memory generation separately. It is `dreaming` while memories for the latest content are queued, being generated, or waiting for a retry, and `done` once they are generated or when the document needed none. Under the default `dreaming: "dynamic"`, `status` can reach `done` minutes before `dreamingStatus` does, so wait on `dreamingStatus` when the next step reads memories or profiles. Legacy `GET /v3/documents/{id}` returned it as a top-level `dreamingStatus`; v5 nests it under `system` on document reads and on document list items. + ### Retrieve a memory and its history ```bash v5 @@ -98,7 +100,7 @@ const { count, errors } = await supermemory.documents.delete("user_1", { ``` -There is no separate processing list. Read `system.status` on each item returned by `supermemory.list(namespace, "documents")`. Chunks come from `supermemory.list(namespace, "chunks")` or `documents.get` with `include: ["chunks"]`. +There is no separate processing list. Read `system.status` and `system.dreamingStatus` on each item returned by `supermemory.list(namespace, "documents")`. Chunks come from `supermemory.list(namespace, "chunks")` or `documents.get` with `include: ["chunks"]`. ### Forget memories @@ -114,6 +116,7 @@ See [memory forgetting](/migration/api-v5-memory-forgetting) for the complete dr ### Verification - Assert requested empty includes are `[]`, while omitted includes are absent. +- Move readers of the legacy top-level `dreamingStatus` to `system.dreamingStatus`, and gate memory or profile reads on it rather than on `system.status`. - Paginate each resource type until `currentPage >= totalPages`; unselected arrays stay empty. - Verify chunk rows contain their parent `documentId`. - Exercise partial document-delete failures and semantic dry runs. diff --git a/apps/docs/snippets/api-v5-sdks.mdx b/apps/docs/snippets/api-v5-sdks.mdx index a22cfec3..1a2ae281 100644 --- a/apps/docs/snippets/api-v5-sdks.mdx +++ b/apps/docs/snippets/api-v5-sdks.mdx @@ -50,7 +50,7 @@ Some v4 methods are gone from the v5 SDK. Most have a v5 way to do the same thin | `memories.updateMemory(...)` | Update the source document with `client.documents.update(namespace, id, { content })`. | | `documents.search(...)` | `client.search(namespace, { query, searchMode: "chunks" })` | | `documents.chunks(id)` | `client.documents.get(namespace, id, { include: ["chunks"] })` | -| `documents.listProcessing()` | `client.list(namespace, "documents")` and read `system.status` on each item | +| `documents.listProcessing()` | `client.list(namespace, "documents")` and read `system.status` and `system.dreamingStatus` on each item | | `containerTags.merge(...)`, `mergeStatus(...)` | `client.namespaces.delete(source, { moveTo: target })`. One source per call; the response carries an `operationId`. | | `memories.forget({ content })` | `client.memories.forgetMatching(namespace, { query, dryRun: true })` to preview, then `memories.forget(namespace, { ids })` |