mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-09-14 23:21:04 +00:00
9975bb37b9
Some checks failed
CI / Python tests / Unit Tests - py3.12 (push) Waiting to run
CI / Python tests / Unit Tests - py3.13 (push) Waiting to run
CI / Windows / CLI smoke - py3.11 (push) Waiting to run
Deploy / Documentation / Build documentation (push) Waiting to run
Deploy / Documentation / deploy (push) Blocked by required conditions
CI / Python packages / Build and verify distributions (push) Waiting to run
CI / Python quality / Pre-commit (push) Waiting to run
CI / Python tests / Unit Tests - py3.11 (push) Waiting to run
Security / CodeQL / Analyze javascript-typescript (push) Waiting to run
Security / CodeQL / Analyze python (push) Waiting to run
CI / Documentation / Test and build documentation (push) Has been cancelled
110 lines
4.9 KiB
Markdown
110 lines
4.9 KiB
Markdown
[中文版 / Chinese version](./README_ZH.md)
|
|
|
|
# LongMemEval Benchmark
|
|
|
|
LongMemEval is a benchmark for **long-term memory over multi-session chat
|
|
histories**. Each item provides a chronologically ordered set of chat sessions
|
|
between a user and an assistant, followed by a probing question whose answer is
|
|
only recoverable by reasoning over the user-owned memory. ReMe ingests the
|
|
sessions into an isolated per-item workspace, answers the question via an
|
|
agentic (ReAct) mode, and scores the answer with an LLM-as-judge.
|
|
|
|
Question types include single-session (user / assistant / preference),
|
|
multi-session reasoning, knowledge update, and temporal reasoning.
|
|
|
|
Install ReMe and the LongMemEval plugin in editable mode from the repository root:
|
|
|
|
```bash
|
|
python -m pip install -e ".[as]"
|
|
reme plugins install ./plugins/lme --editable
|
|
reme plugins install ./plugins/lme-judge --editable
|
|
reme plugins validate lme
|
|
```
|
|
|
|
The runner explicitly enables the installed `lme` plugin and combines its defaults with
|
|
ReMe's built-in `benchmark` preset. Editable installation keeps changes under
|
|
[`plugins/lme`](../../plugins/lme/README.md) visible without reinstalling the plugin.
|
|
Custom application config paths still work through `reme.config` and can use `extends: benchmark`.
|
|
This directory continues to own the runner, evaluation settings, dataset and outputs.
|
|
Model credentials use the environment variables declared by the shared benchmark configuration.
|
|
|
|
## 1. Get the Dataset
|
|
|
|
ReMe uses only the **cleaned-S** split, hosted on HuggingFace:
|
|
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2).
|
|
The download script fetches it via the hf-mirror.com mirror; to use a different
|
|
mirror, modify `BASE_URL` in [`download.py`](./download.py).
|
|
|
|
```bash
|
|
cd benchmark/longmemeval
|
|
python download.py # saves dataset/longmemeval_s_reme_cleaned.json; skips if already present
|
|
```
|
|
|
|
Ground truth is embedded in the data file.
|
|
|
|
## 2. Run
|
|
|
|
From the repository root:
|
|
|
|
```bash
|
|
python benchmark/longmemeval/run.py
|
|
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
|
|
python benchmark/longmemeval/run.py -q # quiet: only eval-level logs
|
|
python benchmark/longmemeval/run.py --log-level WARNING # reduce eval runner logs
|
|
python benchmark/longmemeval/run.py --reme-log-level WARNING # reduce reme internal logs
|
|
python benchmark/longmemeval/run.py --eval_only # reuse existing workspaces, query + judge only
|
|
```
|
|
|
|
## 3. Pipeline
|
|
|
|
1. Load the dataset (ground truth is embedded in the data file).
|
|
2. For each item, create an isolated workspace and ingest sessions in chronological order.
|
|
3. If a custom application configuration enables `auto_dream`, trigger it when sessions cross the configured hour
|
|
(default 23:00). The packaged preset leaves it disabled.
|
|
4. Answer each question via agentic (ReAct) mode.
|
|
5. Judge the answer (binary yes/no) with the `answer_judge` job and print per-type accuracy.
|
|
|
|
## 4. Key config — `benchmark/longmemeval/config.yaml`
|
|
|
|
| Key | Meaning |
|
|
| --- | --- |
|
|
| `dataset.path` | Dataset file to evaluate (e.g. `longmemeval_s_reme_cleaned.json`); ground truth is included. |
|
|
| `dataset.start_index` / `num_items` | Slice of items to evaluate. |
|
|
| `dataset.question_types` | Filter by question type; empty = all. |
|
|
| `dataset.workspace_root` | Per-item workspace root (`benchmark/longmemeval/workspaces/longmemeval-s`). |
|
|
| `evaluation.num_workers` | `0` = auto (cpu-2), `1` = sequential, `>1` = parallel. |
|
|
| `evaluation.filter_future_sessions` | Only ingest sessions with timestamp ≤ `question_date`. |
|
|
| `reme.config` | ReMe config used (`benchmark`). |
|
|
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | Dream triggering behavior. |
|
|
| `output.dir` | Results directory (`benchmark/longmemeval/results`). |
|
|
|
|
## 5. Outputs
|
|
|
|
Results are JSON files written to `output.dir` as `results_<timestamp>.json`,
|
|
with a per-type accuracy summary also printed to the console. Logging
|
|
conventions are shared across benchmarks — see the
|
|
[top-level README](../README.md#outputs--logs).
|
|
|
|
## 6. Reference Results
|
|
|
|
### cleaned-s
|
|
|
|
**Basic settings**
|
|
|
|
1. Modified auto-memory prompt, auto-dream disabled.
|
|
2. All sessions in reme-memory are strictly earlier than the question time.
|
|
|
|
**Results**
|
|
|
|
agentscope==2.0.4.post1, conda reme env, 32 workers, eval-only (reusing prebuilt memory)
|
|
(2026-08-06, 500 items, total 10.0 min)
|
|
|
|
| Type | Agentic | input tok/q | output tok/q | total tok/q | tool calls/q |
|
|
|---|---|---|---|---|---|
|
|
| knowledge-update | 0.910 | 31,581 | 589 | 32,169 | 2.90 |
|
|
| multi-session | 0.842 | 52,837 | 1,474 | 54,311 | 4.21 |
|
|
| single-session-assistant | 1.000 | 15,596 | 279 | 15,875 | 1.89 |
|
|
| single-session-preference | 0.633 | 36,802 | 818 | 37,620 | 3.60 |
|
|
| single-session-user | 0.986 | 27,433 | 359 | 27,792 | 2.60 |
|
|
| temporal-reasoning | 0.902 | 62,674 | 985 | 63,659 | 4.97 |
|
|
| **OVERALL** | **0.894** | **43,448** | **876** | **44,324** | **3.69** |
|