ReMe/benchmark

中文版 / Chinese version

ReMe Benchmarks

Reproduction guide for the two memory benchmarks shipped with ReMe:

• LongMemEval — long-term memory over multi-session chat histories.
• BEAM — memory capability over long-context chat cases with rubric-based judging.

Each benchmark runs its own end-to-end pipeline: ingest sessions into an isolated per-item workspace, answer probing questions via an agentic (ReAct) mode, then score answers with an LLM-as-judge.

1. Prerequisites

Install ReMe with dev + core extras (Python 3.11+):

pip install -e ".[dev,core]"

Configure model credentials in a project-root .env file (copied from example.env). The runners auto-load .env from the repository root. Required variables typically include:

LLM_API_KEY=...
LLM_BASE_URL=...
EMBEDDING_API_KEY=...
EMBEDDING_BASE_URL=...

Model names and component wiring live in the ReMe configs referenced by each benchmark (reme/config/lme.yaml and reme/config/beam.yaml).

2. Download Datasets

Each benchmark keeps its own data under its directory: benchmark/<name>/dataset (input data), benchmark/<name>/workspaces (per-item memory workspaces), and benchmark/<name>/results (evaluation outputs). All three are excluded from Git.

LongMemEval — ReMe uses only the cleaned-S split, hosted on HuggingFace: agentscope-ai/ReMe_longmemeval_clean_s_v2 (the script downloads via the hf-mirror.com mirror; to use a different mirror, modify BASE_URL in download.py):

cd benchmark/longmemeval
python download.py            # saves dataset/longmemeval_s_reme_cleaned.json; skips if already present

BEAM (public repository, cloned into benchmark/beam/dataset/):

mkdir -p benchmark/beam/dataset
cd benchmark/beam/dataset
git clone https://github.com/mohammadtavakoli78/BEAM.git

After cloning, benchmark/beam/dataset/BEAM/ should contain chats/, src/, topics/ and other subdirectories.

3. Run LongMemEval

From the repository root:

python benchmark/longmemeval/run.py
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
python benchmark/longmemeval/run.py -q                        # quiet: only eval-level logs
python benchmark/longmemeval/run.py --log-level WARNING       # reduce eval runner logs
python benchmark/longmemeval/run.py --reme-log-level WARNING  # reduce reme internal logs
python benchmark/longmemeval/run.py --eval_only               # reuse existing workspaces, query + judge only

Pipeline

1. Load the dataset (ground truth is embedded in the data file).
2. For each item, create an isolated workspace and ingest sessions in chronological order.
3. Trigger auto_dream when consecutive sessions cross the configured hour (default 23:00).
4. Answer each question via agentic (ReAct) mode.
5. Judge the answer (binary yes/no) with the answer_judge job and print per-type accuracy.

Key config — benchmark/longmemeval/config.yaml

Key	Meaning
dataset.path	Dataset file to evaluate (e.g. longmemeval_s_reme_cleaned.json); ground truth is included.
dataset.start_index / num_items	Slice of items to evaluate.
dataset.question_types	Filter by question type; empty = all.
dataset.workspace_root	Per-item workspace root (benchmark/longmemeval/workspaces/longmemeval-s).
evaluation.num_workers	0 = auto (cpu-2), 1 = sequential, >1 = parallel.
evaluation.filter_future_sessions	Only ingest sessions with timestamp ≤ question_date.
reme.config	ReMe config used (lme.yaml).
reme.dream_trigger_hour / dream_scan_days / dream_max_units	Dream triggering behavior.
output.dir	Results directory (benchmark/longmemeval/results).

4. Run BEAM

From the repository root:

python benchmark/beam/run.py
python benchmark/beam/run.py --config benchmark/beam/config.yaml
python benchmark/beam/run.py -q                        # quiet
python benchmark/beam/run.py --eval_only               # reuse existing workspaces, query + judge only

Pipeline

1. For each case, load chat.json and convert each batch into a ReMe session.
2. Ingest sessions in chronological order into an isolated workspace, then digest_update.
3. Answer each probing question via agentic (ReAct) mode.
4. Score answers with BEAM's rubric-based answer_judge job and print per-type averages.

Key config — benchmark/beam/config.yaml

Key	Meaning
dataset.beam_root	BEAM dataset root (benchmark/beam/dataset/BEAM).
dataset.chat_size	Variant to run: 100K / 500K / 1M / 10M.
dataset.case_ids	Specific cases (e.g. ["1","2"]); empty = all cases.
dataset.start_index / num_items	Case pagination (num_items 0 = all).
dataset.workspace_root	Per-case workspace root (benchmark/beam/workspaces/beam).
evaluation.num_workers	0 = auto, 1 = sequential, >1 = parallel.
reme.config	ReMe config used (beam.yaml).
output.dir	Results directory (benchmark/beam/results).

5. Outputs & Logs

• Results: JSON files written to output.dir (results_<timestamp>.json for LongMemEval, results_<chat_size>_<timestamp>.json for BEAM). A summary with per-type accuracy/score is also printed to the console.
• Logs: when output.log_to_file is enabled, per-run logs are written to logs/<log_prefix>_<timestamp>/ (a runner.log plus one worker-<pid>.log per worker process).

6. Stopping a Run

Parallel runs spawn a process tree. To terminate a run and all its workers cleanly:

bash benchmark/kill.sh <PID>

The script gracefully sends SIGTERM to the whole process tree, then escalates to SIGKILL for any process that does not exit within 5 seconds.

7. Reference Results

Recorded evaluation results are available in:

• longmemeval.md
• beam.md