ReMe/docs/en/memory_search.md

# Memory Search

Memory Search is ReMe's memory retrieval entry point. The default background loop continuously builds Markdown under
`daily/` and `digest/` into a searchable chunk index and wikilink graph. At query time, it first recalls the most
relevant fragments and then expands context along the bidirectional links of the files containing those fragments.
`reme reindex` has a broader rebuild scope that also scans `resource/` and JSONL; it is intentionally different from the
live watcher.

<p align="center">
  <img src="../figure/auto-index-and-memory-search.svg" alt="ReMe Auto Index and Memory Search indexing, recall, fusion, and link expansion" width="92%">
</p>

For the general semantics of file layers, frontmatter, wikilinks, and chunking, see
[Memory as File](./memory_as_file.md). This page focuses on index maintenance and query execution.

```text
workspace files
  ├─ index_update_loop: detect added / modified / deleted
  ├─ update_index_step: file -> FileNode + FileChunk[]
  ├─ file_store: store chunks, BM25, optional embeddings, and the wikilink graph
  └─ search_step: BM25 / vector recall -> RRF fusion -> link expansion
```

## What It Searches

The default `index_update_loop` watches two memory directories:

- `daily_dir`: daily working memory and session memory cards generated by Auto Memory.
- `digest_dir`: long-term distilled digest nodes.

The live watcher handles only the `md` suffix. A separate `resource_watch_loop` watches `resource_dir`, and Auto
Resource turns those inputs into daily cards that enter the live index. When `reme reindex` is run manually, its
configuration scans
`daily_dir`, `digest_dir`, and `resource_dir` for `md` and `jsonl`; Markdown uses the `markdown` chunker and JSONL uses
the
`jsonl` chunker.

## How the Index Is Built

### Index Update

The background Job `index_update_loop` maintains the index using configuration from `reme/config/default.yaml`:

```yaml
index_update_loop:
  backend: background
  watch_dirs: [daily_dir, digest_dir]
  watch_suffixes: [md]
  steps:
    - backend: init_changes_step
      monitor_type: file_store
      monitor_name: default
      dispatch_steps: [ update_index_step ]
    - backend: watch_changes_step
      dispatch_steps: [ update_index_step ]
```

`init_changes_step` runs at startup. It scans the watched directories, compares file mtimes on disk with
`FileNode.st_mtime` values already stored in `file_store`, calculates added, modified, and deleted changes, and passes
`context["changes"]` to `update_index_step`.

While the service is running, `watch_changes_step` takes over. It uses `watchfiles.awatch()` to watch the same
directories, groups file events within a quiet window, and uses `coalesce_changes()` to collapse repeated events for the
same path into one stable batch of changes.

`update_index_step` performs the actual index writes:

1. Select a file chunker by suffix.
2. Parse the file into one `FileNode` and multiple `FileChunk` objects.
3. For an added or modified file, delete its old chunks before upserting the new chunks.
4. For a deleted file, remove its records from `file_store`, `keyword_index`, and `file_graph`.
5. When changes exist, dump state to `metadata/` so it can be restored on the next startup.

The Markdown chunker parses YAML frontmatter, heading structure, and wikilinks into `FileNode`, `FileChunk`, and
`FileLink`
objects. For detailed chunking rules, see [Memory as File](./memory_as_file.md#memory-chunking).

### Index Optimization

Both BM25 and the FAISS HNSW vector index use tombstone markers instead of physical removal when deleting nodes; too
many tombstones degrade search performance. An idle-time optimization mechanism is built in—the `optimize_index_cron`
scheduled job compacts tombstones and rebuilds indexes during off-peak hours:

```yaml
optimize_index_cron:
  backend: cron
  cron: "0 2 * * *"
  steps:
    - backend: optimize_index_step
```

By default it runs at 2:00 AM daily; adjust the cron expression to customize the schedule.

## What file_store Contains

The default `file_store.default` backend is `local`:

```yaml
file_store:
  default:
    backend: local
    embedding_store: ""
    keyword_index: default
    file_graph: default
```

It combines three kinds of capability:

| Part                    | Default state | Purpose                                                                 |
|-------------------------|---------------|-------------------------------------------------------------------------|
| `file_chunks`           | Enabled       | Store `FileChunk` text, line numbers, scores, and optional embeddings.  |
| `keyword_index.default` | Enabled       | BM25 inverted index where chunk ID is the document ID.                  |
| `file_graph.default`    | Enabled       | Store `FileNode` objects and wikilink edges.                            |
| `embedding_store`       | Disabled      | When enabled, generate embeddings for chunks and support vector recall. |

Out of the box, search therefore uses primarily BM25 plus link expansion. After setting `embedding_store: default`,
`SearchStep` runs vector and keyword recall together. Additionally, switching the `file_store` `backend` from `local` to
`faiss` upgrades vector retrieval from a linear scan to a FAISS HNSW index, offering faster recall at scale.

The embedding store accepts `health_check_timeout` for its startup probe. A temporary failure skips the current vector
backfill while keeping BM25 available; a later successful provider request resumes the missing-vector backfill
automatically.

Embedded integrations that have already verified a provider can call `resume_embedding(verified=True)`. When changing
the embedding vector space, pass `rebuild=True`; persisted vectors are invalidated before a serial background rebuild,
and vector search remains unavailable until the rebuilt vectors are safely persisted.

## How to Search

The `search` Job is also configured in `default.yaml`:

```yaml
search:
  backend: base
  description: "Hybrid workspace search (vector + BM25, RRF-fused)."
  parameters:
    query: string
    limit: integer
    min_score: number
    start_date: string
    end_date: string
  steps:
    - backend: search_step
      vector_weight: 0.7
      candidate_multiplier: 5.0
      expand_links: true
      max_links_per_direction: 10
```

Call it with:

```bash
reme search query="recent discussions about indexing" limit=5
```

Use `start_date` and `end_date` for inclusive `YYYY-MM-DD` filtering:

```bash
reme search query="index regression" start_date=2026-06-01 end_date=2026-06-20 limit=10
```

`search_step` executes in this order:

```mermaid
flowchart LR
    A["query + limit"] --> B["candidates = min(200, limit * candidate_multiplier)"]
    B --> C["file_store.vector_search(...)"]
    B --> D["file_store.keyword_search(...)"]
    C --> E["RRF fusion"]
    D --> E
    E --> F["min_score filter"]
    F --> G["truncate to limit"]
    G --> H["expand_links(...)"]
    H --> I["Response.answer + metadata"]
```

If only BM25 has results, the BM25 ranking is returned directly. If only vector search has results, the vector ranking
is returned directly. When both have results, they are fused with RRF. RRF does not compare BM25 and cosine scores
directly; it compares ranks in the two result lists:

```text
fused_score = vector_weight / (60 + vector_rank)
            + keyword_weight / (60 + keyword_rank)
```

The default `vector_weight=0.7` gives semantic recall more weight when embeddings are enabled, while keyword search can
still promote chunks with exact term matches.

## How BM25 Works

`keyword_search()` calls `keyword_index.retrieve(query, limit)`. Each chunk is a document in the BM25 index:

- `doc_id` is `FileChunk.id`.
- `content` is `FileChunk.text`.
- The tokenizer splits text into tokens.
- The inverted index records which chunks contain each token and its term frequency within each chunk.
- A query scores only the posting lists matching its tokens and returns the highest-scoring chunk IDs.

When a file changes, `LocalFileStore.upsert()` first removes the BM25 documents corresponding to the file's old
`chunk_ids`
and then adds the new chunk text. Deletion is lazy; the index can later be compacted with optimize.

## Progressive Expansion

"Progressive" in Memory Search does not mean putting the entire repository into one result. Retrieval expands in three
layers:

1. Chunk recall: return only the `limit` most relevant text fragments.
2. File location: each result includes `path:start_line-end_line`. Pass the path and line bounds separately as `path`,
   `start_line`, and `end_line` when calling `read`; the range is not part of the `path` value.
3. Link neighbors: call `expand_links()` for each matched file and expand at most `max_links_per_direction` outlinks and
   inlinks.

Expansion data comes from `file_graph` rather than rescanning files:

```text
matched chunk
  -> chunk.path
  -> file_store.get_outlinks(path)
  -> file_store.get_inlinks(path)
  -> file_store.get_nodes(neighbor_paths)
  -> render neighbor path, name, description, and anchor
```

This keeps search results short while still showing which long-term nodes, resources, or other daily notes a memory
connects to. If a result is worth pursuing, use `read path=...` to open the source or
`traverse path=... depth=2` to continue along the wikilink graph.

## Return Format

`SearchStep` writes results in two places:

- `response.answer`: human-readable text. Each matched block contains its path, line numbers, score, and chunk content,
  followed by outlinks and inlinks.
- `response.metadata`: structured programmatic results containing `results`, `link_expansion`, and `counts`.

Typical text structure:

```text
========== daily/2026-06-20/retrieval-regression.md:12-28 [score=0.0317 keyword=4.8120] ==========
...matched memory fragment...
  outlinks (2):
    -> digest/indexing.md  name="Indexing"  description="..."
  inlinks (1):
    <- daily/2026-06-19.md  name="..."
```

`counts` reports how many vector and keyword candidates were recalled and how many results were ultimately returned.
With embeddings disabled by default, `vector` is usually `0` and `hybrid` is `false`.