mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-11-02 07:13:50 +00:00
99421ec211
- Add compressor job and compressor as_llm component to reme/config/beam.yaml (aligned with lme.yaml) so that compress_session: true works for BEAM - Add graceful degradation guard in search_v2._compress_session_entries: when the compressor job is missing from the active config, log a warning and skip compression instead of raising 'Job compressor not found'. Skipped when there is no app_context so unit tests mocking run_job still drive compression behavior. - Fix BEAM download instructions in README.md/README_ZH.md: add mkdir -p before cd benchmark/beam/dataset (the directory is gitignored and absent in a fresh clone)
155 lines
6 KiB
Markdown
155 lines
6 KiB
Markdown
[中文版 / Chinese version](./README_ZH.md)
|
|
|
|
# ReMe Benchmarks
|
|
|
|
Reproduction guide for the two memory benchmarks shipped with ReMe:
|
|
|
|
- **LongMemEval** — long-term memory over multi-session chat histories.
|
|
- **BEAM** — memory capability over long-context chat cases with rubric-based judging.
|
|
|
|
Each benchmark runs its own end-to-end pipeline: ingest sessions into an isolated
|
|
per-item workspace, answer probing questions via an agentic (ReAct) mode,
|
|
then score answers with an LLM-as-judge.
|
|
|
|
## 1. Prerequisites
|
|
|
|
Install ReMe with dev + core extras (Python 3.11+):
|
|
|
|
```bash
|
|
pip install -e ".[dev,core]"
|
|
```
|
|
|
|
Configure model credentials in a project-root `.env` file (copied from `example.env`).
|
|
The runners auto-load `.env` from the repository root. Required variables typically include:
|
|
|
|
```bash
|
|
LLM_API_KEY=...
|
|
LLM_BASE_URL=...
|
|
EMBEDDING_API_KEY=...
|
|
EMBEDDING_BASE_URL=...
|
|
```
|
|
|
|
Model names and component wiring live in the ReMe configs referenced by each benchmark
|
|
(`reme/config/lme.yaml` and `reme/config/beam.yaml`).
|
|
|
|
## 2. Download Datasets
|
|
|
|
Each benchmark keeps its own data under its directory:
|
|
`benchmark/<name>/dataset` (input data), `benchmark/<name>/workspaces`
|
|
(per-item memory workspaces), and `benchmark/<name>/results` (evaluation outputs).
|
|
All three are excluded from Git.
|
|
|
|
**LongMemEval** — ReMe uses only the **cleaned-S** split, hosted on HuggingFace:
|
|
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)
|
|
(the script downloads via the hf-mirror.com mirror; to use a different mirror,
|
|
modify `BASE_URL` in `download.py`):
|
|
|
|
```bash
|
|
cd benchmark/longmemeval
|
|
python download.py # saves dataset/longmemeval_s_reme_cleaned.json; skips if already present
|
|
```
|
|
|
|
**BEAM** (public repository, cloned into `benchmark/beam/dataset/`):
|
|
|
|
```bash
|
|
mkdir -p benchmark/beam/dataset
|
|
cd benchmark/beam/dataset
|
|
git clone https://github.com/mohammadtavakoli78/BEAM.git
|
|
```
|
|
|
|
After cloning, `benchmark/beam/dataset/BEAM/` should contain `chats/`, `src/`,
|
|
`topics/` and other subdirectories.
|
|
|
|
## 3. Run LongMemEval
|
|
|
|
From the repository root:
|
|
|
|
```bash
|
|
python benchmark/longmemeval/run.py
|
|
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
|
|
python benchmark/longmemeval/run.py -q # quiet: only eval-level logs
|
|
python benchmark/longmemeval/run.py --log-level WARNING # reduce eval runner logs
|
|
python benchmark/longmemeval/run.py --reme-log-level WARNING # reduce reme internal logs
|
|
python benchmark/longmemeval/run.py --eval_only # reuse existing workspaces, query + judge only
|
|
```
|
|
|
|
### Pipeline
|
|
|
|
1. Load the dataset (ground truth is embedded in the data file).
|
|
2. For each item, create an isolated workspace and ingest sessions in chronological order.
|
|
3. Trigger `auto_dream` when consecutive sessions cross the configured hour (default 23:00).
|
|
4. Answer each question via agentic (ReAct) mode.
|
|
5. Judge the answer (binary yes/no) with the `answer_judge` job and print per-type accuracy.
|
|
|
|
### Key config — `benchmark/longmemeval/config.yaml`
|
|
|
|
| Key | Meaning |
|
|
| --- | --- |
|
|
| `dataset.path` | Dataset file to evaluate (e.g. `longmemeval_s_reme_cleaned.json`); ground truth is included. |
|
|
| `dataset.start_index` / `num_items` | Slice of items to evaluate. |
|
|
| `dataset.question_types` | Filter by question type; empty = all. |
|
|
| `dataset.workspace_root` | Per-item workspace root (`benchmark/longmemeval/workspaces/longmemeval-s`). |
|
|
| `evaluation.num_workers` | `0` = auto (cpu-2), `1` = sequential, `>1` = parallel. |
|
|
| `evaluation.filter_future_sessions` | Only ingest sessions with timestamp ≤ `question_date`. |
|
|
| `reme.config` | ReMe config used (`lme.yaml`). |
|
|
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | Dream triggering behavior. |
|
|
| `output.dir` | Results directory (`benchmark/longmemeval/results`). |
|
|
|
|
## 4. Run BEAM
|
|
|
|
From the repository root:
|
|
|
|
```bash
|
|
python benchmark/beam/run.py
|
|
python benchmark/beam/run.py --config benchmark/beam/config.yaml
|
|
python benchmark/beam/run.py -q # quiet
|
|
python benchmark/beam/run.py --eval_only # reuse existing workspaces, query + judge only
|
|
```
|
|
|
|
### Pipeline
|
|
|
|
1. For each case, load `chat.json` and convert each batch into a ReMe session.
|
|
2. Ingest sessions in chronological order into an isolated workspace, then `digest_update`.
|
|
3. Answer each probing question via agentic (ReAct) mode.
|
|
4. Score answers with BEAM's rubric-based `answer_judge` job and print per-type averages.
|
|
|
|
### Key config — `benchmark/beam/config.yaml`
|
|
|
|
| Key | Meaning |
|
|
| --- | --- |
|
|
| `dataset.beam_root` | BEAM dataset root (`benchmark/beam/dataset/BEAM`). |
|
|
| `dataset.chat_size` | Variant to run: `100K` / `500K` / `1M` / `10M`. |
|
|
| `dataset.case_ids` | Specific cases (e.g. `["1","2"]`); empty = all cases. |
|
|
| `dataset.start_index` / `num_items` | Case pagination (`num_items` `0` = all). |
|
|
| `dataset.workspace_root` | Per-case workspace root (`benchmark/beam/workspaces/beam`). |
|
|
| `evaluation.num_workers` | `0` = auto, `1` = sequential, `>1` = parallel. |
|
|
| `reme.config` | ReMe config used (`beam.yaml`). |
|
|
| `output.dir` | Results directory (`benchmark/beam/results`). |
|
|
|
|
## 5. Outputs & Logs
|
|
|
|
- **Results**: JSON files written to `output.dir`
|
|
(`results_<timestamp>.json` for LongMemEval,
|
|
`results_<chat_size>_<timestamp>.json` for BEAM). A summary with per-type
|
|
accuracy/score is also printed to the console.
|
|
- **Logs**: when `output.log_to_file` is enabled, per-run logs are written to
|
|
`logs/<log_prefix>_<timestamp>/` (a `runner.log` plus one `worker-<pid>.log`
|
|
per worker process).
|
|
|
|
## 6. Stopping a Run
|
|
|
|
Parallel runs spawn a process tree. To terminate a run and all its workers cleanly:
|
|
|
|
```bash
|
|
bash benchmark/kill.sh <PID>
|
|
```
|
|
|
|
The script gracefully sends `SIGTERM` to the whole process tree, then escalates to
|
|
`SIGKILL` for any process that does not exit within 5 seconds.
|
|
|
|
## 7. Reference Results
|
|
|
|
Recorded evaluation results are available in:
|
|
|
|
- [`longmemeval.md`](./results_md/longmemeval.md)
|
|
- [`beam.md`](./results_md/beam.md)
|