Find a file
xyf2020 4eb2adf961
feat(faiss_file_store): upgrade FAISS to HNSW index with async reindex (#390)
* feat(file_store): upgrade FAISS to HNSW index with async reindex and path constraint

- Replace IndexFlatIP with IndexHNSWFlat for better recall/speed tradeoff
- Add dynamic efSearch (limit * 5) scaled to query request size
- Add async_reindex option: background rebuild with generation-based invalidation
- Extract _delete_nodes() in LocalFileStore for subclass reuse
- Add unit tests for file store consistency

* fix: resolve pylint warnings in faiss store and test file

* refactor(file_store): replace generation-based reindex with event-flag worker

- Replace _reindex_generation/lock/task with a single long-lived worker
  coroutine consuming an asyncio.Event flag; repeated submissions coalesce
- Use local index reference in vector_search to avoid TOCTOU on self._faiss_index
- Pass index explicitly to _set_ef_search for consistency
- Track _index_writes to re-arm reindex after concurrent writes
- Update tests to match new internal API

* fix: resolve pylint too-many-return-statements and implicit-booleaness warnings

* feat(file_store): add refine maintenance hook and incremental embedding backfill

- Add refine() idle-time maintenance hook to BaseFileStore/LocalFileStore
- FaissLocalFileStore: incremental vector add on backfill instead of full rebuild
- Dynamic tombstone compaction threshold scaled by index size
- Add RefineStoreStep with daily cron job (refine_store_cron)
- Enable faiss backend and embedding_store by default in default.yaml
- Add unit tests for faiss index maintenance

* chore(deps): promote faiss-cpu to core dependencies

faiss backend is now the default file_store, so faiss-cpu moves from
the optional [core] extra to the base dependencies list.

* feat: rename refine_store to optimize_index and add vecdb_path_constraint

- Rename refine_store step to optimize_index with cron job scheduling
- Add vecdb_path_constraint to file_store components
- Update default.yaml with optimize_index_cron and faiss backend comment
- Update memory_search docs (en/zh) for FAISS vector management
- Update unit tests for index maintenance

* feat(faiss): add embedding digest to reject stale sidecar after partial dump

Add _chunks_embedding_digest() that computes an order-independent SHA-256
over (chunk_id, float16 embedding) pairs. The digest is written into the
idmap sidecar at dump time and verified at load time. A mismatch means the
sidecar vectors belong to a different chunk generation than the authoritative
JSONL — detectable even when the live-ID set is unchanged (same-ID in-place
update crash window).

Add test_faiss_rejects_stale_sidecar_after_partial_dump reproducing the
crash-between-writes scenario and asserting digest-based rejection.

Compress verbose docstrings/comments in existing tests for pylint line
budget.

---------

Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
2026-07-27 19:54:38 +08:00
.github/workflows feat(index): add bounded memory-aware batch processing (#381) 2026-07-20 17:25:00 +08:00
benchmark feat(search): scoped dedup, session-chunk merge, and unified recall formatting (#384) 2026-07-22 17:17:23 +08:00
cookbook feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
docs feat(faiss_file_store): upgrade FAISS to HNSW index with async reindex (#390) 2026-07-27 19:54:38 +08:00
plugins feat(plugins): add Hermes Agent memory provider (#365) 2026-07-17 14:06:12 +08:00
reme feat(faiss_file_store): upgrade FAISS to HNSW index with async reindex (#390) 2026-07-27 19:54:38 +08:00
skills feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
tests feat(faiss_file_store): upgrade FAISS to HNSW index with async reindex (#390) 2026-07-27 19:54:38 +08:00
.gitignore feat(faiss_file_store): upgrade FAISS to HNSW index with async reindex (#390) 2026-07-27 19:54:38 +08:00
.pre-commit-config.yaml feat: add daily paper cookbook and DingTalk agent integration (#385) 2026-07-22 19:17:01 +08:00
AGENTS.md refactor(agent): unify agent subprocess env, sessions, skills, and MCP/service jobs (#382) 2026-07-20 23:52:14 +08:00
CLAUDE.md docs: restructure documentation and update content organization (#339) 2026-07-13 16:50:46 +08:00
example.env feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
LICENSE feat(reme_ai): implement memory retrieval and merging functionality 2025-08-25 16:10:53 +08:00
pyproject.toml feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
README.md feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00
README_ZH.md feat: add Auto Fin cookbook and managed outbound proxy support (#392) 2026-07-25 18:09:39 +08:00

ReMe Logo

Python Version PyPI Version PyPI Downloads GitHub commit activity License English 简体中文 GitHub Stars DeepWiki

agentscope-ai%2FReMe | Trendshift

An agent memory layer that turns conversations and resources into readable, editable, searchable Markdown memory.

Previous versions: 0.3.x · 0.2.x · MemoryScope

🧠 ReMe is a local-first memory layer for AI agents. It turns conversations and resources into file-based long-term memory, then continuously indexes, links, and consolidates that memory for future recall.

Core Ideas

  • Memory as File: Markdown files with frontmatter and wikilinks serve as memory nodes that both users and agents can read and write directly.
  • Self-evolving knowledge base: Auto Memory, Auto Resource, and Auto Dream progressively transform conversations and resources into long-term memories, while automatically building wikilink relationships.
  • Progressive hybrid search: ReMe combines wikilinks, BM25, and embeddings for hybrid retrieval across keyword matching, semantic recall, and relationship expansion.
  • Agent-friendly integration: SKILL.md + CLI integration makes it easy for different agents to read, write, maintain, and reuse memory.

ReMe Design Philosophy

🔭 Use Cases

  • Personal assistants: Give personal assistants such as QwenPaw, OpenClaw, and Hermes a user-editable long-term memory layer.
  • Coding agents: Preserve coding style, project background, repository decisions, and workflow experience across sessions when integrating with coding agents such as Claude Code.
  • LLM Wiki: Turn conversations, notes, and resources into a searchable, traceable, and linked Markdown knowledge base that both users and agents can maintain.
  • Self-evolving agents: Support agents that learn from experience by saving successful paths, failed attempts, reusable procedures, and periodic reflections as memory.

📰 News

🚀 Quick Start

Installation

ReMe requires Python 3.11+.

Install from pip:

pip install "reme-ai[core]"

Install from source:

git clone https://github.com/agentscope-ai/ReMe.git
cd ReMe
pip install -e ".[core]"

Environment Variables

Configure environment variables when you want LLM-powered memory evolution or embedding retrieval. Embeddings are disabled by default, so the default setup does not start an embedding model or require an embedding API key.

cat > .env <<'EOF'
# Optional: used only after embedding components are explicitly enabled in the config.
# EMBEDDING_API_KEY=sk-xxx
# EMBEDDING_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1

# Required for auto_memory, auto_resource, and auto_dream.
LLM_API_KEY=sk-xxx
LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
EOF

Basic file operations, BM25 search, wikilink traversal, and reading proactive topics can run without LLM credentials.

Note

To enable embedding-based semantic retrieval, uncomment components.as_embedding and components.embedding_store in reme/config/default.yaml, then change components.file_store.default.embedding_store from "" to default. See the memory search guide for details.

Start the Service

reme start

The default service address is 127.0.0.1:2333. If the port is occupied, specify another port:

reme start service.port=8181
# reme start workspace_dir=/tmp/reme-demo service.port=8181

After startup, check the service status. If you use a custom port, replace 2333 in the URL below with that port.

reme version
curl -s http://127.0.0.1:2333/version -H 'Content-Type: application/json' -d '{}'

5-Minute Memory Demo

With the service running, write a memory node, let ReMe index it, then retrieve it:

reme write \
  path=digest/wiki/quick-start-demo \
  name="Quick Start Demo" \
  description="A first ReMe memory node" \
  content="# Quick Start Demo

ReMe stores agent memory as readable Markdown.

Related: [[digest/wiki/memory-as-file.md]]"

reme search query="agent memory markdown" limit=5
reme read path=digest/wiki/quick-start-demo start_line=1 end_line=20

The generated file is ordinary Markdown with frontmatter:

---
name: Quick Start Demo
description: A first ReMe memory node
---

# Quick Start Demo

ReMe stores agent memory as readable Markdown.

Related: [[digest/wiki/memory-as-file.md]]

🧑‍🍳 Cookbooks

Cookbooks are optional, end-to-end workflows assembled from ReMe jobs and steps. They are not enabled by the default configuration; select the cookbook's standalone configuration when starting ReMe. Each new cookbook will be added as another row in this table.

Cookbook Capability
Daily Paper Discover and rank papers, analyze PDFs with an agent, and generate file-native notes and a five-minute brief.
Auto Fin Match CLS events to liquid ETFs, study historical reactions, and generate file-native research reports.

📁 Memory System

Memory as File, File as Memory.

ReMe treats memory as files, progressively processing raw conversations and external resources from session/ and resource/ into daily/, then consolidating them into reusable long-term memory nodes under digest/.

Directory Structure

<workspace_dir>/
├── metadata/       # Persistent system state such as indexes, graphs, and catalogs
├── session/        # Raw conversations and agent sessions
│   ├── dialog/
│   │   └── <session_id>.jsonl
│   ├── agentscope/
│   └── claude_code/
├── resource/            # External raw materials
│   └── YYYY-MM-DD/
│       └── <resource>.<ext>
├── daily/               # Lightly processed memory: daily facts, conversation summaries, resource readings
│   ├── YYYY-MM-DD.md
│   └── YYYY-MM-DD/
│       ├── <session_event>.md
│       ├── <resource_stem>.md
│       └── interests.yaml
└── digest/              # Long-term memory: personal facts, procedural experience, knowledge nodes
    ├── personal/
    │   └── {topic/event}.md
    ├── procedure/
    │   └── {topic/event}.md
    └── wiki/
        └── {topic/event}.md

ReMe file-based memory system overview

🧭 Memory Design Philosophy

Capture raw dialogs and resources, refine them into long-term preferences, reusable experience, and valuable knowledge, while keeping the result editable by humans and agents.

Automatic Memory Flow

ReMe follows a capture → index → consolidate → recall loop. Conversations and resources first become daily memory cards; background jobs keep files searchable; auto_dream distills stable knowledge into digest/; agents recall memory through search, wikilinks, or proactive topics.

Capability Entry point What it does Output
auto_memory Agent hook or reme auto_memory Distills useful conversation facts while preserving the raw session. session/dialog/*.jsonl, daily/<date>/<session>.md
auto_resource Resource watcher or reme auto_resource Turns files under resource/<date>/ into source-linked daily cards. daily/<date>/<resource-card>.md
auto_index Background watcher or reme reindex Maintains chunks, the BM25 index, the wikilink graph, and the optional embedding index. Searchable daily/, digest/, and resource/ content
auto_dream dream_cron or reme auto_dream Consolidates changed daily cards into long-term personal, procedure, and wiki memory. digest/**, daily/<date>/interests.yaml
proactive reme proactive before an agent decides to act Reads topics generated by auto_dream; the host agent decides whether and how to mention them. Structured topics from daily/<date>/interests.yaml
Memory as File Auto Memory and Resource
Auto Dream and Proactive Auto Index and Memory Search

🤝 Agent-friendly Integration

ReMe runs as a local memory service and offers multiple integration paths: CLI, HTTP API, MCP server, and SDK. Different agents can choose the path that fits their runtime while sharing the same local memory workspace.

Agents Recommended path What works out of the box
QwenPaw Embed ReMe via the Python SDK. Reuse the app's own lifecycle and model config while keeping memory local and file-based.
Claude Code Start ReMe as an MCP service and install plugins/reme. MCP recall tools, a reme-memory skill, and a Stop hook that records sessions automatically.
Other CLI-capable agents (OpenClaw/Hermes/Codex) Copy or install skills/reme_memory/SKILL.md. Search/read/write memory and call auto_memory, auto_dream, and proactive via the CLI.

Integration demos

Auto Memory Auto Dream
QwenPaw QwenPaw Auto Memory demo QwenPaw Auto Dream demo
Claude Code Claude Code Auto Memory demo Claude Code Auto Dream demo

🛠️ ReMe Operations

ReMe operates the workspace through a unified job interface exposed by the CLI. Agents usually only need retrieval, reading, writing, editing, and automatic memory commands. Lower-level indexing, frontmatter, and file operation commands are mainly for maintenance, debugging, or advanced integration. Run reme help for the full job list.

Command Purpose
reme start Start the local ReMe service.
reme version / reme health_check Check package and component status.
reme status Show stateful data-component memory estimates and process RSS.
reme search Retrieve memory with BM25 and wikilinks by default, plus vectors when enabled.
reme read / reme write / reme edit Inspect and maintain Markdown memory files.
reme auto_memory Turn conversation messages into daily memory cards. Requires LLM credentials.
reme auto_resource Interpret files under resource/ into daily resource cards. Requires LLM credentials.
reme auto_dream / reme proactive Consolidate daily memory into long-term digest and surface topics worth attention.
reme reindex Rebuild search and wikilink indexes from existing files.

🤝 Community and Support

  • Issues and requests: Check Open Issues first. If there is no related discussion, open a new issue with background, expected behavior, and impact scope.
  • Code contributions: Before making changes, read the contribution guide. Source, schemas, and tests are the authoritative architecture and extension guide.
  • Documentation contributions: Submit user-facing documentation changes to the unified documentation repository under reme/<version>/{en,zh}/.
  • Commit convention: Conventional Commits are recommended, for example feat(search): add link expansion option or docs(zh): update quick start.
  • Pre-submit checks: Before submitting a PR, try to run pre-commit run --all-files and pytest. If tests that depend on LLMs, embeddings, or external services cannot run, explain that in the PR.
  • Get help: Use GitHub Issues for bugs and feature requests. Project documentation is available at https://docs.agentscope.io/.

Contributors

Thanks to everyone who has contributed to ReMe:

Contributors

📄 Citation

@software{ReMe2026,
  title = {Remember me, Refine me: Memory Management Kit for Agents},
  author = {ReMe Team},
  url = {https://reme.agentscope.io},
  year = {2026}
}

⚖️ License

This project is open source under the Apache License 2.0. See LICENSE for details.