* feat(benchmark): add golden answer validation and session review for LongMemEval - Introduce GoldenCheckStep to validate LongMemEval golden answers using structured verdicts - Add SessionReviewStep to extract query/answer-relevant evidence from all sessions - Implement concurrent session processing with configurable concurrency limits - Create check_golden job configuration with lme_review and lme_judge agent wrappers - Add Qwen3.7-plus model configuration for enhanced processing capabilities - Include python_execute tool integration for agent-based reasoning and date validation - Generate comprehensive JSON output with session summaries and validation verdicts - Add run_check_golden.py script for batch processing across all LongMemEval samples - Configure proper logging initialization with console and file output options - Update component registry and file I/O modules to support new benchmark features * feat(scripts): add script to summarize LongMemEval check_golden verdicts - Parse check_golden.json files across all LongMemEval samples - Calculate accuracy metrics for golden answers and session IDs - Provide breakdown by question type with percentage calculations - Add command line options for listing bad samples and JSON output - Include progress tracking showing completed vs pending samples - Display confidence scores and date sanity checks statistics * refactor(benchmark): move golden check scripts to longmemeval directory - Moved run_check_golden.py from scripts/ to benchmark/longmemeval/ - Moved stats_check_golden.py from scripts/ to benchmark/longmemeval/ - Updated path resolution to use parents[2] instead of parent.parent - Added new --list-run-failed option to stats script - Added logging directory constant and functions for tracking launched samples - Enhanced stats output with launched count and run failure information - Improved error reporting with run failure details and log file paths * feat(benchmark): add LongMemEval agentic answer workflow with session extraction - Add LmeAgenticAnswerStep, LmeAutoMemoryStep, and LmeExtractSessionStep to __init__.py - Create shared helper render_with_source for displaying search results with session_id - Implement agentic_answer step with vector_search, bm25_search, and extract_session_by_id tools - Add auto_memory step to convert each session into search-friendly daily notes - Create extract_session step to retrieve and analyze raw session content by session_id - Update jinli_lme.yaml with auto_memory, vector_search, bm25_search, and agentic_answer jobs - Configure lme_memory, lme_extract, and lme_agentic_answer agent wrappers - Enhance search steps with include_source option to show session_id metadata - Add proper session_id tracking and collision handling in daily note generation * feat(benchmark): add LongMemEval agentic answer evaluation pipeline - Add session_id tracking to agentic_answer.py result metadata - Introduce run_agentic_answer.py driver for complete pipeline execution - Implement auto_memory, update_index, and agentic_answer job orchestration - Add concurrent execution with configurable limits and staggering - Create aggregation script for collecting tool-call trails and results - Add stats_agentic_answer.py for comprehensive result analysis - Implement resume capability with existing output detection - Generate aggregate.json with per-sample breakdown and tool call summaries * feat(steps): add ClearPathsStep for cleaning workspace outputs before rebuild - Introduce ClearPathsStep to remove stale workspace files/directories - Add support for specifying paths and config_keys as targets to clear - Implement safety checks to prevent deletion of files outside workspace - Add logging for cleared paths and warnings for invalid paths - Configure clear_paths_step in jinli_lme.yaml to clean daily_dir - Add clear_paths_step to clean mem_answer.json before rebuilds * feat(benchmark): add resume functionality to agentic answer runner - Replace --force flag with --resume flag for controlling job execution - By default every job reruns with clean rebuild behavior using config clear steps - Add --resume option to skip samples whose output already exists and continue interrupted batches - Update documentation to reflect new default clean rebuild behavior - Modify job skipping logic to honor resume flag instead of force flag - Update dry-run output to show correct todo jobs based on resume status - Change default example command to use --resume for continuing interrupted runs * feat(benchmark): generate JSONL output for check golden records - Add write_check_golden_list function to create JSONL file - Write all readable check_golden records as JSONL format - Include check_golden_list path in stats output - Display generated JSONL file path in summary report - Maintain UTF-8 encoding with non-ASCII character support * refactor(benchmark): rename answer judge step and integrate LME LLM judge - Rename AnswerJudgeStep to LmeLlmJudgeStep and update imports - Add new llm_judge configuration in jinli_lme.yaml - Update run_agentic_answer.py to include llm_judge in pipeline - Modify LmeLlmJudgeStep to read from query.json and answer.json - Write LLM judgement results back to mem_answer.json - Add command line options for start/end sample range selection - Update aggregate.json generation to include LLM judgement data - Add resume capability for llm_judge job based on judgement presence * refactor(benchmark): rename answer judge step and integrate LME LLM judge - Rename AnswerJudgeStep to LmeLlmJudgeStep and update imports - Add new llm_judge configuration in jinli_lme.yaml - Update run_agentic_answer.py to include llm_judge in pipeline - Modify LmeLlmJudgeStep to read from query.json and answer.json - Write LLM judgement results back to mem_answer.json - Add command line options for start/end sample range selection - Update aggregate.json generation to include LLM judgement data - Add resume capability for llm_judge job based on judgement presence * feat(steps): add wait_for_paths_step to block until workspace files exist - Introduce WaitForPathsStep class that polls for required workspace-relative paths - Add step registration with 'wait_for_paths_step' backend identifier - Implement path validation to ensure targets are within workspace boundaries - Add polling mechanism with configurable intervals via poll_seconds parameter - Include logging functionality with log_every_seconds parameter for status updates - Add metadata tracking of waited paths and duration in response object - Register step in index module and expose in public API - Configure step in jinli_lme.yaml to wait for session_review.json before golden check - Add script rename from run_check_golden.py to run_golden_check.py with enhanced options * feat(benchmark): enhance longmemeval benchmarking with concurrency and progress tracking - Add benchmark extra dependency group with portalocker requirement - Introduce concurrent execution support for golden_check and session_review workflows - Add progress reporting interval option with real-time status updates - Implement global throttling mechanism for session review requests using file locks - Enhance golden check validation with current schema verification - Add active task tracking and graceful shutdown handling - Rename check_golden scripts to golden_check for consistency - Update statistics reporting with correct/incorrect terminology instead of reasonable - Add stale format detection and compatibility handling for verdict fields - Include both_correct rate calculation in accuracy metrics - Add concurrency and staggering options for better resource management * ci(workflow): add Windows smoke test workflow - Create new workflow file .github/workflows/windows-smoke.yml - Configure workflow to trigger on push and pull request events - Set up Python environment with version 3.11 - Install package dependencies using pip - Run version job as smoke test for CLI functionality - Enable concurrency control to prevent duplicate runs - Use matrix strategy for Python version testing * feat(benchmark): add retry mechanism and health check for session review - Added retry configuration options (retry_initial_seconds, retry_max_seconds, retry_max_attempts) to jinli_lme.yaml - Implemented exponential backoff retry logic with configurable parameters in session_review step - Added output_is_healthy function to verify session_review.json integrity and absence of failed reviews - Updated resume functionality to skip only healthy outputs instead of all existing files - Integrated JSON parsing and validation to check for failed reviews in output files - Enhanced error handling and logging for retry attempts and recovery scenarios * feat(benchmark): add LongMemEval session review statistics script - Create stats_session_review.py to summarize session_review.json artifacts - Add command line options for listing failed, missing, and run failed samples - Implement JSON output mode for programmatic consumption - Calculate and display health statistics including total samples, healthy outputs, failed sessions - Provide detailed failure information with session IDs and error messages - Generate re-run commands for samples with failed reviews - Add percentage calculations for better statistical overview - Include support for multiple output formats and detailed logging * feat(benchmark): add LongMemEval output cleanup script and enhance golden check retry logic - Added clean_sample_outputs.py script to remove generated LongMemEval files while preserving source inputs - Implemented configurable retry mechanism in golden_check.py with exponential backoff strategy - Added retry parameters (initial/max seconds and max attempts) to control failure recovery behavior - Integrated asyncio support for asynchronous sleep during retry intervals - Configured default retry settings in jinli_lme.yaml with 5s initial and 300s maximum intervals - Preserved core files (query.json, answer.json, session/) while cleaning generated artifacts * feat(benchmark): add AppleDouble file cleanup to sample output cleaner - Remove AppleDouble files starting with '._' recursively including under session/ - Add is_under helper function to check if path is inside parent directory - Track targets in set to avoid duplicate processing - Include AppleDouble files in cleanup targets when not already covered by existing targets - Maintain dry-run mode as default behavior with --apply flag for actual deletion * refactor(benchmark): update LongMemEval sample output cleaning script - Add time and Iterator imports for enhanced functionality - Add --progress-every argument to control progress reporting frequency - Replace is_under function with iter_sample_targets generator - Implement detailed progress tracking with timing measurements - Add sample-by-sample processing with elapsed time reporting - Include AppleDouble file detection within session directory - Update target counting and deletion statistics display - Add conditional progress updates based on progress-every setting - Improve dry-run mode with would-delete indication * chore(benchmark): increase initial interval for session review step - Changed START_INTERVAL_SECONDS from 1.0 to 3.0 seconds - Adjusted timing parameters for better benchmark stability * refactor(benchmark): implement coordinated retry mechanism for session reviews - Add retry gate condition to coordinate concurrent review attempts - Implement wait_for_healthy_start_slot to handle sequential retries - Create mark_retrying and mark_recovered functions to track retry states - Update reply_with_retry to accept index parameter for coordination - Add has_prior_retry logic to prevent race conditions during recovery - Ensure proper cleanup of retry state on success or failure - Maintain backward compatibility while adding coordination features * chore(benchmark): adjust session review start interval timeout - Changed START_INTERVAL_SECONDS from 3.0 to 5.0 seconds - Increased initial delay for session review benchmark step - Updated timeout configuration for improved stability * refactor(benchmark): update session review concurrency and throttling mechanism - Replace global throttle with per-process concurrency control - Add concurrency parameter with default value of 30 in config - Add start_interval_seconds parameter with default value of 2 seconds - Change default concurrency from 3 to 1 in command line interface - Update documentation to reflect new throttling behavior - Implement semaphore-based concurrency limiting for review tasks - Modify retry mechanism to use local locking instead of global files - Remove portalocker dependency for cross-process throttling * refactor(config): update session review configuration and concurrency settings - Removed deprecated retry configuration parameters from jinli_lme.yaml - Increased MAX_CONCURRENCY from 30 to 60 in session_review.py - Reduced START_INTERVAL_SECONDS from 2.0 to 1.0 in session_review.py - Cleaned up redundant backend specifications in configuration file - Simplified agent wrapper configurations by removing obsolete retry settings * feat(benchmark): enhance LME auto memory step with advanced scheduling and error handling - Add datetime parsing functionality for LongMemEval timestamps with regex pattern - Implement configurable concurrency limits with MAX_CONCURRENCY of 60 - Introduce retry mechanism with exponential backoff for agent interactions - Add session filtering based on date comparison with question_date validation - Create rate limiting with start interval control between requests - Implement sophisticated retry coordination using asyncio conditions - Add comprehensive error tracking for failed and filtered session extracts - Remove deprecated concurrency parameter from jinli_lme.yaml configuration - Add structured output validation in session review step - Include detailed metadata reporting with session statistics and errors * fix(benchmark): adjust default concurrency for auto_memory job - Changed default concurrency from 3 to 1 for auto_memory job to prevent API overload - Updated help text to reflect new default value of 1 for concurrency parameter - Modified documentation to clarify concurrency behavior varies by job type * refactor(search): replace hardcoded candidate multiplier with constant - Introduced _CANDIDATE_MULTIPLIER constant set to 10 - Replaced hardcoded factor of 5 with _CANDIDATE_MULTIPLIER in BM25 search - Replaced hardcoded factor of 5 with _CANDIDATE_MULTIPLIER in vector search - Updated test to verify both search steps use ten times limit for candidates - Imported VectorSearchStep and Bm25SearchStep in test module - Added comprehensive test case for candidate count calculation logic * feat(lme): add data inspection error handling with fallback mechanism - Implemented non-retryable data inspection error markers detection - Added _is_data_inspection_error method to identify inspection failures - Created fallback handling for data inspection errors in auto memory extraction - Added fallback handling for data inspection errors in session review - Extended failed extracts tracking with non-retryable and fallback flags - Separated fallback extracts from regular failed extracts in reporting - Enhanced error logging with specific data inspection failure messages - Updated metrics to track fallback extractions and reviews separately - Maintained existing retry logic for other exception types * feat(benchmark): enhance session review statistics with fallback tracking - Add support for identifying and listing non-retryable fallback reviews - Introduce --list-fallback argument to display fallback review details - Separate retryable failures from non-retryable fallbacks in reporting - Track fallback samples and sessions separately from failed ones - Update console output to show both retryable and non-retryable categories - Include fallback details in JSON output with reasons and session info - Modify failure counting logic to distinguish between retryable and fallback reviews * feat(benchmark): add question_id tracking and enhanced fallback reporting - Add question_id function to extract query.question_id from data - Initialize question_id_by_id dictionary to store question IDs by index - Store question_id for each sample during data processing - Enhance fallback output to include question IDs and session information - Format sample labels with question IDs when available - Display session IDs associated with each fallback case * feat(benchmark): add question_id support and improve bad sample reporting - Add question_id_for function to extract question_id from multiple sources - Add sample_label function to format samples as idx(question_id) when available - Store question_id in data dictionary during processing - Change bad_golden and bad_sessions to store full records instead of just indices - Update list_bad output to show formatted labels with question_id information - Improve error reporting with more detailed sample identification * feat(benchmark): enhance golden check stats with structured output - Add related_session_ids function to extract session IDs from verdict records - Create grouped_records function to group records by question type - Replace flat list output with JSON-formatted grouped records in list_bad option - Replace flat list output with JSON-formatted grouped records in list_bad_sessions option - Maintain Chinese labels while adding structured data presentation - Improve readability of bad verdict record display with hierarchical grouping * feat(benchmark): update data structure for question indexing - Replace sample_label with _idx field for index tracking - Add question_id field to store _question_id values - Maintain backward compatibility with empty string defaults - Preserve existing session_id functionality - Update data mapping to include new fields in grouped results * refactor(benchmark): streamline golden answer verification process - Replace relevance filtering with comprehensive information extraction - Remove is_relevant field and simplify session summary structure - Change relevant_info to extracted_info for clarity - Update golden check logic to work with full extractions instead of filtered summaries - Simplify prompt instructions to focus on complete information extraction - Remove redundant schema validation and structured output requirements - Adjust statistics calculation to match new extraction approach - Update metadata field names to reflect extraction rather than relevance checking * feat(benchmark): add selective file deletion option to clean_sample_outputs - Add --filename argument to delete only specific root-level files - Modify iter_sample_targets function to accept optional filenames filter - Implement validation for root-level filename constraints - Update function calls to pass filenames parameter - Add example usage for selective file deletion in documentation * feat(benchmark): add error count metrics to golden check statistics - Added golden_bad, session_bad, and both_bad calculation fields - Updated console output format to include error counts per question type - Modified table display to show both accuracy rates and error numbers - Enhanced statistical summary with additional error breakdown metrics * test(search): update search step tests with include_source parameter - Added include_source=False parameter to VectorSearchStep initialization - Added include_source=False parameter to Bm25SearchStep initialization - Maintained existing RuntimeContext parameters for both search steps - Updated test calls to match new constructor signature with include_source option |
||
|---|---|---|
| .github/workflows | ||
| benchmark/longmemeval | ||
| docs | ||
| plugins | ||
| reme | ||
| skills | ||
| tests | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| AGENTS.md | ||
| CLAUDE.md | ||
| example.env | ||
| LICENSE | ||
| pyproject.toml | ||
| README.md | ||
| README_ZH.md | ||
An agent memory layer that turns conversations and resources into readable, editable, searchable Markdown memory.
Previous versions: 0.3.x · 0.2.x · MemoryScope
🧠 ReMe is a local-first memory layer for AI agents. It turns conversations and resources into file-based long-term memory, then continuously indexes, links, and consolidates that memory for future recall.
✨ Core Ideas
- Memory as File: Markdown files with frontmatter and wikilinks serve as memory nodes that both users and agents can read and write directly.
- Self-evolving knowledge base: Auto Memory, Auto Resource, and Auto Dream progressively transform conversations and resources into long-term memories, while automatically building wikilink relationships.
- Progressive hybrid search: ReMe combines wikilinks, BM25, and embeddings for hybrid retrieval across keyword matching, semantic recall, and relationship expansion.
- Agent-friendly integration: SKILL.md + CLI integration makes it easy for different agents to read, write, maintain, and reuse memory.
🔭 Use Cases
- Personal assistants: Give personal assistants such as QwenPaw, OpenClaw, and Hermes a user-editable long-term memory layer.
- Coding agents: Preserve coding style, project background, repository decisions, and workflow experience across sessions when integrating with coding agents such as Claude Code.
- LLM Wiki: Turn conversations, notes, and resources into a searchable, traceable, and linked Markdown knowledge base that both users and agents can maintain.
- Self-evolving agents: Support agents that learn from experience by saving successful paths, failed attempts, reusable procedures, and periodic reflections as memory.
📰 News
- [2026.07] - Our paper Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution has been accepted to Findings of ACL 2026.
🚀 Quick Start
Installation
ReMe requires Python 3.11+.
Install from pip:
pip install "reme-ai[core]"
Install from source:
git clone https://github.com/agentscope-ai/ReMe.git
cd ReMe
pip install -e ".[core]"
Environment Variables
Configure environment variables when you want LLM-powered memory evolution or embedding retrieval. Embeddings are disabled by default, so the default setup does not start an embedding model or require an embedding API key.
cat > .env <<'EOF'
# Optional: used only after embedding components are explicitly enabled in the config.
# EMBEDDING_API_KEY=sk-xxx
# EMBEDDING_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
# Required for auto_memory, auto_resource, and auto_dream.
LLM_API_KEY=sk-xxx
LLM_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1
EOF
Basic file operations, BM25 search, wikilink traversal, and reading proactive topics can run without LLM credentials.
Note
To enable embedding-based semantic retrieval, uncomment
components.as_embeddingandcomponents.embedding_storeinreme/config/default.yaml, then changecomponents.file_store.default.embedding_storefrom""todefault. See the memory search guide for details.
Start the Service
reme start
The default service address is 127.0.0.1:2333. If the port is occupied, specify another port:
reme start service.port=8181
# reme start workspace_dir=/tmp/reme-demo service.port=8181
After startup, check the service status. If you use a custom port, replace 2333 in the URL below with that port.
reme version
curl -s http://127.0.0.1:2333/version -H 'Content-Type: application/json' -d '{}'
5-Minute Memory Demo
With the service running, write a memory node, let ReMe index it, then retrieve it:
reme write \
path=digest/wiki/quick-start-demo \
name="Quick Start Demo" \
description="A first ReMe memory node" \
content="# Quick Start Demo
ReMe stores agent memory as readable Markdown.
Related: [[digest/wiki/memory-as-file.md]]"
reme search query="agent memory markdown" limit=5
reme read path=digest/wiki/quick-start-demo start_line=1 end_line=20
The generated file is ordinary Markdown with frontmatter:
---
name: Quick Start Demo
description: A first ReMe memory node
---
# Quick Start Demo
ReMe stores agent memory as readable Markdown.
Related: [[digest/wiki/memory-as-file.md]]
📁 Memory System
Memory as File, File as Memory.
ReMe treats memory as files, progressively processing raw conversations and external resources from session/ and
resource/ into daily/, then consolidating them into reusable long-term memory nodes under digest/.
Directory Structure
<workspace_dir>/
├── metadata/ # Persistent system state such as indexes, graphs, and catalogs
├── session/ # Raw conversations and agent sessions
│ ├── dialog/
│ │ └── <session_id>.jsonl
│ ├── agentscope/
│ └── claude_code/
├── resource/ # External raw materials
│ └── YYYY-MM-DD/
│ └── <resource>.<ext>
├── daily/ # Lightly processed memory: daily facts, conversation summaries, resource readings
│ ├── YYYY-MM-DD.md
│ └── YYYY-MM-DD/
│ ├── <session_event>.md
│ ├── <resource_stem>.md
│ └── interests.yaml
└── digest/ # Long-term memory: personal facts, procedural experience, knowledge nodes
├── personal/
│ └── {topic/event}.md
├── procedure/
│ └── {topic/event}.md
└── wiki/
└── {topic/event}.md
🧭 Memory Design Philosophy
Capture raw dialogs and resources, refine them into long-term preferences, reusable experience, and valuable knowledge, while keeping the result editable by humans and agents.
Automatic Memory Flow
ReMe follows a capture → index → consolidate → recall loop. Conversations and resources first become daily memory cards;
background jobs keep files searchable; auto_dream distills stable knowledge into digest/; agents recall memory
through search, wikilinks, or proactive topics.
| Capability | Entry point | What it does | Output |
|---|---|---|---|
auto_memory |
Agent hook or reme auto_memory |
Distills useful conversation facts while preserving the raw session. | session/dialog/*.jsonl, daily/<date>/<session>.md |
auto_resource |
Resource watcher or reme auto_resource |
Turns files under resource/<date>/ into source-linked daily cards. |
daily/<date>/<resource-card>.md |
auto_index |
Background watcher or reme reindex |
Maintains chunks, the BM25 index, the wikilink graph, and the optional embedding index. | Searchable daily/, digest/, and resource/ content |
auto_dream |
dream_cron or reme auto_dream |
Consolidates changed daily cards into long-term personal, procedure, and wiki memory. | digest/**, daily/<date>/interests.yaml |
proactive |
reme proactive before an agent decides to act |
Reads topics generated by auto_dream; the host agent decides whether and how to mention them. |
Structured topics from daily/<date>/interests.yaml |
|
|
|
|
|
|
🤝 Agent-friendly Integration
ReMe runs as a local memory service and offers multiple integration paths: CLI, HTTP API, MCP server, and SDK. Different agents can choose the path that fits their runtime while sharing the same local memory workspace.
| Agents | Recommended path | What works out of the box |
|---|---|---|
| QwenPaw | Embed ReMe via the Python SDK. | Reuse the app's own lifecycle and model config while keeping memory local and file-based. |
| Claude Code | Start ReMe as an MCP service and install plugins/reme. | MCP recall tools, a reme-memory skill, and a Stop hook that records sessions automatically. |
| Other CLI-capable agents (OpenClaw/Hermes/Codex) | Copy or install skills/reme_memory/SKILL.md. | Search/read/write memory and call auto_memory, auto_dream, and proactive via the CLI. |
Integration demos
| Auto Memory | Auto Dream | |
| QwenPaw |
|
|
| Claude Code |
|
|
🛠️ ReMe Operations
ReMe operates the workspace through a unified job interface exposed by the CLI. Agents usually only need retrieval,
reading, writing, editing, and automatic memory commands. Lower-level indexing, frontmatter, and file operation commands
are mainly for maintenance, debugging, or advanced integration. Run reme help for the full job list.
| Command | Purpose |
|---|---|
reme start |
Start the local ReMe service. |
reme version / reme health_check |
Check package and component status. |
reme search |
Retrieve memory with BM25 and wikilinks by default, plus vectors when enabled. |
reme read / reme write / reme edit |
Inspect and maintain Markdown memory files. |
reme auto_memory |
Turn conversation messages into daily memory cards. Requires LLM credentials. |
reme auto_resource |
Interpret files under resource/ into daily resource cards. Requires LLM credentials. |
reme auto_dream / reme proactive |
Consolidate daily memory into long-term digest and surface topics worth attention. |
reme reindex |
Rebuild search and wikilink indexes from existing files. |
🤝 Community and Support
- Issues and requests: Check Open Issues first. If there is no related discussion, open a new issue with background, expected behavior, and impact scope.
- Code contributions: Before making changes, read the contribution guide. Source, schemas, and tests are the authoritative architecture and extension guide.
- Documentation contributions: Submit user-facing documentation changes to the
unified documentation repository under
reme/<version>/{en,zh}/. - Commit convention: Conventional Commits are recommended, for example
feat(search): add link expansion optionordocs(zh): update quick start. - Pre-submit checks: Before submitting a PR, try to run
pre-commit run --all-filesandpytest. If tests that depend on LLMs, embeddings, or external services cannot run, explain that in the PR. - Get help: Use GitHub Issues for bugs and feature requests. Project documentation is available at https://docs.agentscope.io/.
Contributors
Thanks to everyone who has contributed to ReMe:
📄 Citation
@software{ReMe2026,
title = {Remember me, Refine me: Memory Management Kit for Agents},
author = {ReMe Team},
url = {https://reme.agentscope.io},
year = {2026}
}
⚖️ License
This project is open source under the Apache License 2.0. See LICENSE for details.