ReMe/docs/en/memory_search.md

Memory Search

Memory Search is ReMe's memory retrieval entry point. The default background loop continuously builds Markdown under daily/ and digest/ into a searchable chunk index and wikilink graph. At query time, it first recalls the most relevant fragments and then expands context along the bidirectional links of the files containing those fragments. reme reindex has a broader rebuild scope that also scans resource/ and JSONL; it is intentionally different from the live watcher.

For the general semantics of file layers, frontmatter, wikilinks, and chunking, see Memory as File. This page focuses on index maintenance and query execution.

workspace files
  ├─ index_update_loop: detect added / modified / deleted
  ├─ update_index_step: file -> FileNode + FileChunk[]
  ├─ file_store: store chunks, BM25, optional embeddings, and the wikilink graph
  └─ search_step: BM25 / vector recall -> RRF fusion -> link expansion

What It Searches

The default index_update_loop watches two memory directories:

• daily_dir: daily working memory and session memory cards generated by Auto Memory.
• digest_dir: long-term distilled digest nodes.

The live watcher handles only the md suffix. A separate resource_watch_loop watches resource_dir, and Auto Resource turns those inputs into daily cards that enter the live index. When reme reindex is run manually, its configuration scans daily_dir, digest_dir, and resource_dir for md and jsonl; Markdown uses the markdown chunker and JSONL uses the jsonl chunker.

How the Index Is Built

Index Update

The background Job index_update_loop maintains the index using configuration from reme/config/default.yaml:

index_update_loop:
  backend: background
  watch_dirs: [daily_dir, digest_dir]
  watch_suffixes: [md]
  steps:
    - backend: init_changes_step
      monitor_type: file_store
      monitor_name: default
      dispatch_steps: [ update_index_step ]
    - backend: watch_changes_step
      dispatch_steps: [ update_index_step ]

init_changes_step runs at startup. It scans the watched directories, compares file mtimes on disk with FileNode.st_mtime values already stored in file_store, calculates added, modified, and deleted changes, and passes context["changes"] to update_index_step.

While the service is running, watch_changes_step takes over. It uses watchfiles.awatch() to watch the same directories, groups file events within a quiet window, and uses coalesce_changes() to collapse repeated events for the same path into one stable batch of changes.

update_index_step performs the actual index writes:

1. Select a file chunker by suffix.
2. Parse the file into one FileNode and multiple FileChunk objects.
3. For an added or modified file, delete its old chunks before upserting the new chunks.
4. For a deleted file, remove its records from file_store, keyword_index, and file_graph.
5. When changes exist, dump state to metadata/ so it can be restored on the next startup.

The Markdown chunker parses YAML frontmatter, heading structure, and wikilinks into FileNode, FileChunk, and FileLink objects. For detailed chunking rules, see Memory as File.

Index Optimization

Both BM25 and the FAISS HNSW vector index use tombstone markers instead of physical removal when deleting nodes; too many tombstones degrade search performance. An idle-time optimization mechanism is built in—the optimize_index_cron scheduled job compacts tombstones and rebuilds indexes during off-peak hours:

optimize_index_cron:
  backend: cron
  cron: "0 2 * * *"
  steps:
    - backend: optimize_index_step

By default it runs at 2:00 AM daily; adjust the cron expression to customize the schedule.

What file_store Contains

The default file_store.default backend is local:

file_store:
  default:
    backend: local
    embedding_store: ""
    keyword_index: default
    file_graph: default

It combines three kinds of capability:

Part	Default state	Purpose
file_chunks	Enabled	Store FileChunk text, line numbers, scores, and optional embeddings.
keyword_index.default	Enabled	BM25 inverted index where chunk ID is the document ID.
file_graph.default	Enabled	Store FileNode objects and wikilink edges.
embedding_store	Disabled	When enabled, generate embeddings for chunks and support vector recall.

Out of the box, search therefore uses primarily BM25 plus link expansion. After setting embedding_store: default, SearchStep runs vector and keyword recall together. Additionally, switching the file_store backend from local to faiss upgrades vector retrieval from a linear scan to a FAISS HNSW index, offering faster recall at scale.

The embedding store accepts health_check_timeout for its startup probe. A temporary failure skips the current vector backfill while keeping BM25 available; a later successful provider request resumes the missing-vector backfill automatically.

Embedded integrations that have already verified a provider can call resume_embedding(verified=True). When changing the embedding vector space, pass rebuild=True; persisted vectors are invalidated before a serial background rebuild, and vector search remains unavailable until the rebuilt vectors are safely persisted.

How to Search

The search Job is also configured in default.yaml:

search:
  backend: base
  description: "Hybrid workspace search (vector + BM25, RRF-fused)."
  parameters:
    query: string
    limit: integer
    min_score: number
    start_date: string
    end_date: string
  steps:
    - backend: search_step
      vector_weight: 0.7
      candidate_multiplier: 5.0
      expand_links: true
      max_links_per_direction: 10

Call it with:

reme search query="recent discussions about indexing" limit=5

Use start_date and end_date for inclusive YYYY-MM-DD filtering:

reme search query="index regression" start_date=2026-06-01 end_date=2026-06-20 limit=10

search_step executes in this order:

flowchart LR
    A["query + limit"] --> B["candidates = min(200, limit * candidate_multiplier)"]
    B --> C["file_store.vector_search(...)"]
    B --> D["file_store.keyword_search(...)"]
    C --> E["RRF fusion"]
    D --> E
    E --> F["min_score filter"]
    F --> G["truncate to limit"]
    G --> H["expand_links(...)"]
    H --> I["Response.answer + metadata"]

If only BM25 has results, the BM25 ranking is returned directly. If only vector search has results, the vector ranking is returned directly. When both have results, they are fused with RRF. RRF does not compare BM25 and cosine scores directly; it compares ranks in the two result lists:

fused_score = vector_weight / (60 + vector_rank)
            + keyword_weight / (60 + keyword_rank)

The default vector_weight=0.7 gives semantic recall more weight when embeddings are enabled, while keyword search can still promote chunks with exact term matches.

How BM25 Works

keyword_search() calls keyword_index.retrieve(query, limit). Each chunk is a document in the BM25 index:

• doc_id is FileChunk.id.
• content is FileChunk.text.
• The tokenizer splits text into tokens.
• The inverted index records which chunks contain each token and its term frequency within each chunk.
• A query scores only the posting lists matching its tokens and returns the highest-scoring chunk IDs.

When a file changes, LocalFileStore.upsert() first removes the BM25 documents corresponding to the file's old chunk_ids and then adds the new chunk text. Deletion is lazy; the index can later be compacted with optimize.

Progressive Expansion

"Progressive" in Memory Search does not mean putting the entire repository into one result. Retrieval expands in three layers:

1. Chunk recall: return only the limit most relevant text fragments.
2. File location: each result includes path:start_line-end_line. Pass the path and line bounds separately as path, start_line, and end_line when calling read; the range is not part of the path value.
3. Link neighbors: call expand_links() for each matched file and expand at most max_links_per_direction outlinks and inlinks.

Expansion data comes from file_graph rather than rescanning files:

matched chunk
  -> chunk.path
  -> file_store.get_outlinks(path)
  -> file_store.get_inlinks(path)
  -> file_store.get_nodes(neighbor_paths)
  -> render neighbor path, name, description, and anchor

This keeps search results short while still showing which long-term nodes, resources, or other daily notes a memory connects to. If a result is worth pursuing, use read path=... to open the source or traverse path=... depth=2 to continue along the wikilink graph.

Return Format

SearchStep writes results in two places:

• response.answer: human-readable text. Each matched block contains its path, line numbers, score, and chunk content, followed by outlinks and inlinks.
• response.metadata: structured programmatic results containing results, link_expansion, and counts.

Typical text structure:

========== daily/2026-06-20/retrieval-regression.md:12-28 [score=0.0317 keyword=4.8120] ==========
...matched memory fragment...
  outlinks (2):
    -> digest/indexing.md  name="Indexing"  description="..."
  inlinks (1):
    <- daily/2026-06-19.md  name="..."

counts reports how many vector and keyword candidates were recalled and how many results were ultimately returned. With embeddings disabled by default, vector is usually 0 and hybrid is false.