diff --git a/benchmark/README.md b/benchmark/README.md deleted file mode 100644 index 0522b09e..00000000 --- a/benchmark/README.md +++ /dev/null @@ -1,155 +0,0 @@ -[中文版 / Chinese version](./README_ZH.md) - -# ReMe Benchmarks - -Reproduction guide for the two memory benchmarks shipped with ReMe: - -- **LongMemEval** — long-term memory over multi-session chat histories. -- **BEAM** — memory capability over long-context chat cases with rubric-based judging. - -Each benchmark runs its own end-to-end pipeline: ingest sessions into an isolated -per-item workspace, answer probing questions via an agentic (ReAct) mode, -then score answers with an LLM-as-judge. - -## 1. Prerequisites - -Install ReMe with dev + core extras (Python 3.11+): - -```bash -pip install -e ".[dev,core]" -``` - -Configure model credentials in a project-root `.env` file (copied from `example.env`). -The runners auto-load `.env` from the repository root. Required variables typically include: - -```bash -LLM_API_KEY=... -LLM_BASE_URL=... -EMBEDDING_API_KEY=... -EMBEDDING_BASE_URL=... -``` - -Model names and component wiring live in the ReMe configs referenced by each benchmark -(`reme/config/lme.yaml` and `reme/config/beam.yaml`). - -## 2. Download Datasets - -Each benchmark keeps its own data under its directory: -`benchmark//dataset` (input data), `benchmark//workspaces` -(per-item memory workspaces), and `benchmark//results` (evaluation outputs). -All three are excluded from Git. - -**LongMemEval** — ReMe uses only the **cleaned-S** split, hosted on HuggingFace: -[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2) -(the script downloads via the hf-mirror.com mirror; to use a different mirror, -modify `BASE_URL` in `download.py`): - -```bash -cd benchmark/longmemeval -python download.py # saves dataset/longmemeval_s_reme_cleaned.json; skips if already present -``` - -**BEAM** (public repository, cloned into `benchmark/beam/dataset/`): - -```bash -mkdir -p benchmark/beam/dataset -cd benchmark/beam/dataset -git clone https://github.com/mohammadtavakoli78/BEAM.git -``` - -After cloning, `benchmark/beam/dataset/BEAM/` should contain `chats/`, `src/`, -`topics/` and other subdirectories. - -## 3. Run LongMemEval - -From the repository root: - -```bash -python benchmark/longmemeval/run.py -python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml -python benchmark/longmemeval/run.py -q # quiet: only eval-level logs -python benchmark/longmemeval/run.py --log-level WARNING # reduce eval runner logs -python benchmark/longmemeval/run.py --reme-log-level WARNING # reduce reme internal logs -python benchmark/longmemeval/run.py --eval_only # reuse existing workspaces, query + judge only -``` - -### Pipeline - -1. Load the dataset (ground truth is embedded in the data file). -2. For each item, create an isolated workspace and ingest sessions in chronological order. -3. Trigger `auto_dream` when consecutive sessions cross the configured hour (default 23:00). -4. Answer each question via agentic (ReAct) mode. -5. Judge the answer (binary yes/no) with the `answer_judge` job and print per-type accuracy. - -### Key config — `benchmark/longmemeval/config.yaml` - -| Key | Meaning | -| --- | --- | -| `dataset.path` | Dataset file to evaluate (e.g. `longmemeval_s_reme_cleaned.json`); ground truth is included. | -| `dataset.start_index` / `num_items` | Slice of items to evaluate. | -| `dataset.question_types` | Filter by question type; empty = all. | -| `dataset.workspace_root` | Per-item workspace root (`benchmark/longmemeval/workspaces/longmemeval-s`). | -| `evaluation.num_workers` | `0` = auto (cpu-2), `1` = sequential, `>1` = parallel. | -| `evaluation.filter_future_sessions` | Only ingest sessions with timestamp ≤ `question_date`. | -| `reme.config` | ReMe config used (`lme.yaml`). | -| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | Dream triggering behavior. | -| `output.dir` | Results directory (`benchmark/longmemeval/results`). | - -## 4. Run BEAM - -From the repository root: - -```bash -python benchmark/beam/run.py -python benchmark/beam/run.py --config benchmark/beam/config.yaml -python benchmark/beam/run.py -q # quiet -python benchmark/beam/run.py --eval_only # reuse existing workspaces, query + judge only -``` - -### Pipeline - -1. For each case, load `chat.json` and convert each batch into a ReMe session. -2. Ingest sessions in chronological order into an isolated workspace, then `digest_update`. -3. Answer each probing question via agentic (ReAct) mode. -4. Score answers with BEAM's rubric-based `answer_judge` job and print per-type averages. - -### Key config — `benchmark/beam/config.yaml` - -| Key | Meaning | -| --- | --- | -| `dataset.beam_root` | BEAM dataset root (`benchmark/beam/dataset/BEAM`). | -| `dataset.chat_size` | Variant to run: `100K` / `500K` / `1M` / `10M`. | -| `dataset.case_ids` | Specific cases (e.g. `["1","2"]`); empty = all cases. | -| `dataset.start_index` / `num_items` | Case pagination (`num_items` `0` = all). | -| `dataset.workspace_root` | Per-case workspace root (`benchmark/beam/workspaces/beam`). | -| `evaluation.num_workers` | `0` = auto, `1` = sequential, `>1` = parallel. | -| `reme.config` | ReMe config used (`beam.yaml`). | -| `output.dir` | Results directory (`benchmark/beam/results`). | - -## 5. Outputs & Logs - -- **Results**: JSON files written to `output.dir` - (`results_.json` for LongMemEval, - `results__.json` for BEAM). A summary with per-type - accuracy/score is also printed to the console. -- **Logs**: when `output.log_to_file` is enabled, per-run logs are written to - `logs/_/` (a `runner.log` plus one `worker-.log` - per worker process). - -## 6. Stopping a Run - -Parallel runs spawn a process tree. To terminate a run and all its workers cleanly: - -```bash -bash benchmark/kill.sh -``` - -The script gracefully sends `SIGTERM` to the whole process tree, then escalates to -`SIGKILL` for any process that does not exit within 5 seconds. - -## 7. Reference Results - -Recorded evaluation results are available in: - -- [`longmemeval.md`](./results_md/longmemeval.md) -- [`beam.md`](./results_md/beam.md) diff --git a/benchmark/README_ZH.md b/benchmark/README_ZH.md deleted file mode 100644 index 0ac8fd3e..00000000 --- a/benchmark/README_ZH.md +++ /dev/null @@ -1,149 +0,0 @@ -# ReMe 评测复现说明 - -ReMe 内置两个记忆能力评测基准的复现指南： - -- **LongMemEval** —— 面向多轮多会话历史的长期记忆能力评测。 -- **BEAM** —— 面向长上下文对话场景、基于评分细则（rubric）打分的记忆能力评测。 - -每个基准都会运行完整的端到端流程：将会话摄入独立的按条目隔离的工作区， -以 agentic（ReAct）模式回答探测问题， -最后由 LLM-as-judge 对答案进行打分。 - -## 1. 环境准备 - -安装 ReMe 及 dev + core 附加依赖（Python 3.11+）： - -```bash -pip install -e ".[dev,core]" -``` - -在项目根目录配置 `.env`（可从 `example.env` 复制），运行脚本会自动从仓库根目录加载 `.env`。 -通常需要以下变量： - -```bash -LLM_API_KEY=... -LLM_BASE_URL=... -EMBEDDING_API_KEY=... -EMBEDDING_BASE_URL=... -``` - -模型名称与组件装配位于各基准引用的 ReMe 配置中 -（`reme/config/lme.yaml` 与 `reme/config/beam.yaml`）。 - -## 2. 下载数据集 - -每个基准的数据都存放在各自目录下：`benchmark//dataset`（输入数据）、 -`benchmark//workspaces`（按条目隔离的记忆工作区）、 -`benchmark//results`（评测输出）。三者均不纳入 Git 版本管理。 - -**LongMemEval** —— ReMe 仅使用 **cleaned-S** 版本，数据托管在 HuggingFace： -[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2) -（下载脚本经 hf-mirror.com 镜像源获取，如需更换源请修改 `download.py` 中的 `BASE_URL`）： - -```bash -cd benchmark/longmemeval -python download.py # 保存为 dataset/longmemeval_s_reme_cleaned.json，已存在则自动跳过 -``` - -**BEAM**（公开仓库，clone 到 `benchmark/beam/dataset/` 下）： - -```bash -mkdir -p benchmark/beam/dataset -cd benchmark/beam/dataset -git clone https://github.com/mohammadtavakoli78/BEAM.git -``` - -clone 完成后，`benchmark/beam/dataset/BEAM/` 目录下应包含 `chats/`、`src/`、`topics/` 等子目录。 - -## 3. 运行 LongMemEval - -在仓库根目录执行： - -```bash -python benchmark/longmemeval/run.py -python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml -python benchmark/longmemeval/run.py -q # 安静模式：仅评测级日志 -python benchmark/longmemeval/run.py --log-level WARNING # 降低评测 runner 日志 -python benchmark/longmemeval/run.py --reme-log-level WARNING # 降低 reme 内部日志 -python benchmark/longmemeval/run.py --eval_only # 复用已有工作区，仅执行查询 + 评判 -``` - -### 流程 - -1. 加载数据集（ground truth 已内嵌在数据文件中）。 -2. 为每个条目创建独立工作区，按时间顺序摄入会话。 -3. 当相邻会话跨越配置的时刻（默认 23:00）时触发 `auto_dream`。 -4. 以 agentic（ReAct）模式回答每个问题。 -5. 通过 `answer_judge` 任务对答案做二元（yes/no）评判，并输出各类型准确率。 - -### 关键配置 —— `benchmark/longmemeval/config.yaml` - -| 配置项 | 含义 | -| --- | --- | -| `dataset.path` | 待评测的数据集文件（如 `longmemeval_s_reme_cleaned.json`），已包含 ground truth。 | -| `dataset.start_index` / `num_items` | 评测条目的切片范围。 | -| `dataset.question_types` | 按问题类型过滤，空表示全部。 | -| `dataset.workspace_root` | 条目工作区根目录（`benchmark/longmemeval/workspaces/longmemeval-s`）。 | -| `evaluation.num_workers` | `0` = 自动（cpu-2），`1` = 串行，`>1` = 并行。 | -| `evaluation.filter_future_sessions` | 仅摄入时间戳 ≤ `question_date` 的会话。 | -| `reme.config` | 使用的 ReMe 配置（`lme.yaml`）。 | -| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | dream 触发行为。 | -| `output.dir` | 结果目录（`benchmark/longmemeval/results`）。 | - -## 4. 运行 BEAM - -在仓库根目录执行： - -```bash -python benchmark/beam/run.py -python benchmark/beam/run.py --config benchmark/beam/config.yaml -python benchmark/beam/run.py -q # 安静模式 -python benchmark/beam/run.py --eval_only # 复用已有工作区，仅执行查询 + 评判 -``` - -### 流程 - -1. 为每个 case 加载 `chat.json`，将每个 batch 转换为一个 ReMe 会话。 -2. 按时间顺序将会话摄入独立工作区，随后执行 `digest_update`。 -3. 以 agentic（ReAct）模式回答每个探测问题。 -4. 通过 BEAM 基于 rubric 的 `answer_judge` 任务打分，并输出各类型平均分。 - -### 关键配置 —— `benchmark/beam/config.yaml` - -| 配置项 | 含义 | -| --- | --- | -| `dataset.beam_root` | BEAM 数据集根目录（`benchmark/beam/dataset/BEAM`）。 | -| `dataset.chat_size` | 运行的变体：`100K` / `500K` / `1M` / `10M`。 | -| `dataset.case_ids` | 指定 case（如 `["1","2"]`），空表示全部。 | -| `dataset.start_index` / `num_items` | case 分页（`num_items` 为 `0` 表示全部）。 | -| `dataset.workspace_root` | case 工作区根目录（`benchmark/beam/workspaces/beam`）。 | -| `evaluation.num_workers` | `0` = 自动，`1` = 串行，`>1` = 并行。 | -| `reme.config` | 使用的 ReMe 配置（`beam.yaml`）。 | -| `output.dir` | 结果目录（`benchmark/beam/results`）。 | - -## 5. 输出与日志 - -- **结果**：JSON 文件写入 `output.dir` - （LongMemEval 为 `results_.json`， - BEAM 为 `results__.json`）。同时控制台会打印含各类型 - 准确率/分数的汇总。 -- **日志**：当 `output.log_to_file` 开启时，每次运行的日志写入 - `logs/_/`（包含一个 `runner.log` 及每个 worker 进程的 - `worker-.log`）。 - -## 6. 终止运行 - -并行运行会派生进程树。若要干净地终止某次运行及其全部 worker： - -```bash -bash benchmark/kill.sh -``` - -该脚本会先向整个进程树发送 `SIGTERM` 优雅终止，对 5 秒内未退出的进程再升级为 `SIGKILL`。 - -## 7. 参考结果 - -已记录的评测结果见： - -- [`longmemeval.md`](./results_md/longmemeval.md) -- [`beam.md`](./results_md/beam.md) diff --git a/benchmark/beam/README.md b/benchmark/beam/README.md new file mode 100644 index 00000000..aef8a3b7 --- /dev/null +++ b/benchmark/beam/README.md @@ -0,0 +1,124 @@ +[中文版 / Chinese version](./README_ZH.md) + +# BEAM Benchmark + +BEAM is a benchmark for **memory capability over long-context chat cases**. Each +case contains a very long chat history split into batches; ReMe converts each +batch into a session, ingests them in chronological order, then answers probing +questions via an agentic (ReAct) mode. Answers are scored with BEAM's +rubric-based `answer_judge` job, which produces both a graded score and a binary +verdict, and per-type averages are reported. + +BEAM ships dataset variants by chat size — `100K` / `500K` / `1M` / `10M` — so +memory systems can be stressed at different context lengths. Question types +include abstention, contradiction resolution, event ordering, information +extraction, instruction following, knowledge update, multi-session reasoning, +preference following, summarization, and temporal reasoning. + +> For the shared setup (dependencies, credentials, log conventions) see the +> [top-level benchmark README](../README.md). + +## 1. Get the Dataset + +BEAM is a public repository, cloned into `benchmark/beam/dataset/`: + +```bash +mkdir -p benchmark/beam/dataset +cd benchmark/beam/dataset +git clone https://github.com/mohammadtavakoli78/BEAM.git +``` + +After cloning, `benchmark/beam/dataset/BEAM/` should contain `chats/`, `src/`, +`topics/` and other subdirectories. + +## 2. Run + +From the repository root: + +```bash +python benchmark/beam/run.py +python benchmark/beam/run.py --config benchmark/beam/config.yaml +python benchmark/beam/run.py -q # quiet +python benchmark/beam/run.py --eval_only # reuse existing workspaces, query + judge only +``` + +## 3. Pipeline + +1. For each case, load `chat.json` and convert each batch into a ReMe session. +2. Ingest sessions in chronological order into an isolated workspace, then `digest_update`. +3. Answer each probing question via agentic (ReAct) mode. +4. Score answers with BEAM's rubric-based `answer_judge` job and print per-type averages. + +## 4. Key config — `benchmark/beam/config.yaml` + +| Key | Meaning | +| --- | --- | +| `dataset.beam_root` | BEAM dataset root (`benchmark/beam/dataset/BEAM`). | +| `dataset.chat_size` | Variant to run: `100K` / `500K` / `1M` / `10M`. | +| `dataset.case_ids` | Specific cases (e.g. `["1","2"]`); empty = all cases. | +| `dataset.start_index` / `num_items` | Case pagination (`num_items` `0` = all). | +| `dataset.workspace_root` | Per-case workspace root (`benchmark/beam/workspaces/beam`). | +| `evaluation.num_workers` | `0` = auto, `1` = sequential, `>1` = parallel. | +| `reme.config` | ReMe config used (`beam.yaml`). | +| `output.dir` | Results directory (`benchmark/beam/results`). | + +## 5. Outputs + +Results are JSON files written to `output.dir` as +`results__.json`, with a per-type score summary also +printed to the console. Logging conventions are shared across benchmarks — see +the [top-level README](../README.md#outputs--logs). + +## 6. Reference Results + +> The results below use the longmemeval-version prompt. + +### 100K + +agentscope==2.0.4.post1, conda reme env, 20 workers, eval-only (reusing prebuilt memory) +(2026-08-05, 20 cases / 400 Qs, total 46.0 min) + +| Type | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q | +|---|---|---|---|---|---|---| +| abstention | 0.550 | 0.550 | 96,031 | 1,070 | 97,101 | 4.58 | +| contradiction_resolution | 0.438 | 0.412 | 32,263 | 872 | 33,135 | 2.48 | +| event_ordering | 0.501 | 0.423 | 140,195 | 5,163 | 145,358 | 4.70 | +| information_extraction | 0.873 | 0.832 | 50,245 | 883 | 51,128 | 3.15 | +| instruction_following | 0.750 | 0.725 | 37,986 | 848 | 38,834 | 2.67 | +| knowledge_update | 0.688 | 0.675 | 31,198 | 651 | 31,849 | 2.27 | +| multi_session_reasoning | 0.626 | 0.584 | 85,038 | 4,563 | 89,601 | 4.28 | +| preference_following | 0.925 | 0.912 | 34,281 | 989 | 35,270 | 2.50 | +| summarization | 0.623 | 0.461 | 89,657 | 2,056 | 91,713 | 4.12 | +| temporal_reasoning | 0.637 | 0.625 | 34,563 | 1,049 | 35,612 | 2.52 | +| **OVERALL** | **0.661** | **0.620** | **63,146** | **1,814** | **64,960** | **3.33** | + +Memory Construction average token consumption (default agent, full build over 20 cases): + +| Agent | input tok/case | output tok/case | total tok/case | +|---|---|---|---| +| default | 2,172,316 | 136,697 | 2,309,013 | + +### 1M + +agentscope==2.0.4.post1, conda reme env, 20 workers, full memory build +(2026-08-05, 35 cases / 700 Qs, total 459.2 min) + +| Type | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q | +|---|---|---|---|---|---|---| +| abstention | 0.429 | 0.429 | 118,707 | 1,178 | 119,886 | 4.20 | +| contradiction_resolution | 0.391 | 0.364 | 49,787 | 810 | 50,597 | 2.50 | +| event_ordering | 0.558 | 0.456 | 201,514 | 3,889 | 205,403 | 4.79 | +| information_extraction | 0.809 | 0.772 | 78,950 | 894 | 79,844 | 3.00 | +| instruction_following | 0.852 | 0.832 | 55,757 | 924 | 56,681 | 2.81 | +| knowledge_update | 0.779 | 0.771 | 45,981 | 665 | 46,646 | 2.37 | +| multi_session_reasoning | 0.658 | 0.612 | 138,133 | 2,873 | 141,006 | 4.40 | +| preference_following | 0.798 | 0.777 | 51,796 | 920 | 52,716 | 2.53 | +| summarization | 0.693 | 0.537 | 158,794 | 2,905 | 161,700 | 4.44 | +| temporal_reasoning | 0.536 | 0.536 | 100,176 | 3,148 | 103,324 | 3.90 | +| **OVERALL** | **0.650** | **0.609** | **99,959** | **1,821** | **101,780** | **3.49** | + +Memory Construction average token consumption (default agent, full build over 35 cases): + +| Agent | input tok/case | output tok/case | total tok/case | +|---|---|---|---| +| default | 31,943,817 | 1,417,061 | 33,360,878 | diff --git a/benchmark/beam/README_ZH.md b/benchmark/beam/README_ZH.md new file mode 100644 index 00000000..f562193b --- /dev/null +++ b/benchmark/beam/README_ZH.md @@ -0,0 +1,119 @@ +# BEAM 评测 + +[English version](./README.md) + +BEAM 是一个面向**长上下文对话场景**的记忆能力评测基准。每个 case 包含一段被切分为多个 +batch 的超长对话；ReMe 将每个 batch 转换为一个会话，按时间顺序摄入后，以 agentic（ReAct） +模式回答探测问题。答案由 BEAM 基于 rubric 的 `answer_judge` 任务打分，同时给出分级分数与二元 +判定，并输出各类型平均分。 + +BEAM 按对话规模提供多种数据变体 —— `100K` / `500K` / `1M` / `10M`，可在不同上下文长度下 +压测记忆系统。题型包括 abstention（拒答）、contradiction resolution（矛盾消解）、event +ordering（事件排序）、information extraction（信息抽取）、instruction following（指令遵循）、 +knowledge update（知识更新）、multi-session reasoning（多会话推理）、preference following +（偏好遵循）、summarization（摘要）与 temporal reasoning（时间推理）。 + +> 公共设置（依赖、凭据、日志约定）见[总评测说明](../README_ZH.md)。 + +## 1. 获取数据集 + +BEAM 是公开仓库，clone 到 `benchmark/beam/dataset/` 下： + +```bash +mkdir -p benchmark/beam/dataset +cd benchmark/beam/dataset +git clone https://github.com/mohammadtavakoli78/BEAM.git +``` + +clone 完成后，`benchmark/beam/dataset/BEAM/` 目录下应包含 `chats/`、`src/`、`topics/` 等子目录。 + +## 2. 运行 + +在仓库根目录执行： + +```bash +python benchmark/beam/run.py +python benchmark/beam/run.py --config benchmark/beam/config.yaml +python benchmark/beam/run.py -q # 安静模式 +python benchmark/beam/run.py --eval_only # 复用已有工作区，仅执行查询 + 评判 +``` + +## 3. 流程 + +1. 为每个 case 加载 `chat.json`，将每个 batch 转换为一个 ReMe 会话。 +2. 按时间顺序将会话摄入独立工作区，随后执行 `digest_update`。 +3. 以 agentic（ReAct）模式回答每个探测问题。 +4. 通过 BEAM 基于 rubric 的 `answer_judge` 任务打分，并输出各类型平均分。 + +## 4. 关键配置 —— `benchmark/beam/config.yaml` + +| 配置项 | 含义 | +| --- | --- | +| `dataset.beam_root` | BEAM 数据集根目录（`benchmark/beam/dataset/BEAM`）。 | +| `dataset.chat_size` | 运行的变体：`100K` / `500K` / `1M` / `10M`。 | +| `dataset.case_ids` | 指定 case（如 `["1","2"]`），空表示全部。 | +| `dataset.start_index` / `num_items` | case 分页（`num_items` 为 `0` 表示全部）。 | +| `dataset.workspace_root` | case 工作区根目录（`benchmark/beam/workspaces/beam`）。 | +| `evaluation.num_workers` | `0` = 自动，`1` = 串行，`>1` = 并行。 | +| `reme.config` | 使用的 ReMe 配置（`beam.yaml`）。 | +| `output.dir` | 结果目录（`benchmark/beam/results`）。 | + +## 5. 输出 + +结果以 JSON 文件写入 `output.dir`，文件名为 `results__.json`， +同时控制台会打印含各类型分数的汇总。日志约定在各基准间通用，见 +[总说明](../README_ZH.md#输出与日志)。 + +## 6. 参考结果 + +> 以下结果使用 longmemeval 版本的 prompt。 + +### 100K + +agentscope==2.0.4.post1，conda reme 环境，20 并发，eval-only（复用已构建 memory） +（2026-08-05，20 cases / 400 Qs，总耗时 46.0 min） + +| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q | +|---|---|---|---|---|---|---| +| abstention | 0.550 | 0.550 | 96,031 | 1,070 | 97,101 | 4.58 | +| contradiction_resolution | 0.438 | 0.412 | 32,263 | 872 | 33,135 | 2.48 | +| event_ordering | 0.501 | 0.423 | 140,195 | 5,163 | 145,358 | 4.70 | +| information_extraction | 0.873 | 0.832 | 50,245 | 883 | 51,128 | 3.15 | +| instruction_following | 0.750 | 0.725 | 37,986 | 848 | 38,834 | 2.67 | +| knowledge_update | 0.688 | 0.675 | 31,198 | 651 | 31,849 | 2.27 | +| multi_session_reasoning | 0.626 | 0.584 | 85,038 | 4,563 | 89,601 | 4.28 | +| preference_following | 0.925 | 0.912 | 34,281 | 989 | 35,270 | 2.50 | +| summarization | 0.623 | 0.461 | 89,657 | 2,056 | 91,713 | 4.12 | +| temporal_reasoning | 0.637 | 0.625 | 34,563 | 1,049 | 35,612 | 2.52 | +| **OVERALL** | **0.661** | **0.620** | **63,146** | **1,814** | **64,960** | **3.33** | + +Memory Construction 平均 token 消耗（default agent，20 cases 全量构建）： + +| Agent | input tok/case | output tok/case | total tok/case | +|---|---|---|---| +| default | 2,172,316 | 136,697 | 2,309,013 | + +### 1M + +agentscope==2.0.4.post1，conda reme 环境，20 并发，全量构建 memory +（2026-08-05，35 cases / 700 Qs，总耗时 459.2 min） + +| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q | +|---|---|---|---|---|---|---| +| abstention | 0.429 | 0.429 | 118,707 | 1,178 | 119,886 | 4.20 | +| contradiction_resolution | 0.391 | 0.364 | 49,787 | 810 | 50,597 | 2.50 | +| event_ordering | 0.558 | 0.456 | 201,514 | 3,889 | 205,403 | 4.79 | +| information_extraction | 0.809 | 0.772 | 78,950 | 894 | 79,844 | 3.00 | +| instruction_following | 0.852 | 0.832 | 55,757 | 924 | 56,681 | 2.81 | +| knowledge_update | 0.779 | 0.771 | 45,981 | 665 | 46,646 | 2.37 | +| multi_session_reasoning | 0.658 | 0.612 | 138,133 | 2,873 | 141,006 | 4.40 | +| preference_following | 0.798 | 0.777 | 51,796 | 920 | 52,716 | 2.53 | +| summarization | 0.693 | 0.537 | 158,794 | 2,905 | 161,700 | 4.44 | +| temporal_reasoning | 0.536 | 0.536 | 100,176 | 3,148 | 103,324 | 3.90 | +| **OVERALL** | **0.650** | **0.609** | **99,959** | **1,821** | **101,780** | **3.49** | + +Memory Construction 平均 token 消耗（default agent，35 cases 全量构建）： + +| Agent | input tok/case | output tok/case | total tok/case | +|---|---|---|---| +| default | 31,943,817 | 1,417,061 | 33,360,878 | diff --git a/benchmark/kill.sh b/benchmark/beam/kill.sh similarity index 100% rename from benchmark/kill.sh rename to benchmark/beam/kill.sh diff --git a/benchmark/longmemeval/README.md b/benchmark/longmemeval/README.md new file mode 100644 index 00000000..070f921e --- /dev/null +++ b/benchmark/longmemeval/README.md @@ -0,0 +1,96 @@ +[中文版 / Chinese version](./README_ZH.md) + +# LongMemEval Benchmark + +LongMemEval is a benchmark for **long-term memory over multi-session chat +histories**. Each item provides a chronologically ordered set of chat sessions +between a user and an assistant, followed by a probing question whose answer is +only recoverable by reasoning over the user-owned memory. ReMe ingests the +sessions into an isolated per-item workspace, answers the question via an +agentic (ReAct) mode, and scores the answer with an LLM-as-judge. + +Question types include single-session (user / assistant / preference), +multi-session reasoning, knowledge update, and temporal reasoning. + +> For the shared setup (dependencies, credentials, log conventions) see the +> [top-level benchmark README](../README.md). + +## 1. Get the Dataset + +ReMe uses only the **cleaned-S** split, hosted on HuggingFace: +[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2). +The download script fetches it via the hf-mirror.com mirror; to use a different +mirror, modify `BASE_URL` in [`download.py`](./download.py). + +```bash +cd benchmark/longmemeval +python download.py # saves dataset/longmemeval_s_reme_cleaned.json; skips if already present +``` + +Ground truth is embedded in the data file. + +## 2. Run + +From the repository root: + +```bash +python benchmark/longmemeval/run.py +python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml +python benchmark/longmemeval/run.py -q # quiet: only eval-level logs +python benchmark/longmemeval/run.py --log-level WARNING # reduce eval runner logs +python benchmark/longmemeval/run.py --reme-log-level WARNING # reduce reme internal logs +python benchmark/longmemeval/run.py --eval_only # reuse existing workspaces, query + judge only +``` + +## 3. Pipeline + +1. Load the dataset (ground truth is embedded in the data file). +2. For each item, create an isolated workspace and ingest sessions in chronological order. +3. Trigger `auto_dream` when consecutive sessions cross the configured hour (default 23:00). +4. Answer each question via agentic (ReAct) mode. +5. Judge the answer (binary yes/no) with the `answer_judge` job and print per-type accuracy. + +## 4. Key config — `benchmark/longmemeval/config.yaml` + +| Key | Meaning | +| --- | --- | +| `dataset.path` | Dataset file to evaluate (e.g. `longmemeval_s_reme_cleaned.json`); ground truth is included. | +| `dataset.start_index` / `num_items` | Slice of items to evaluate. | +| `dataset.question_types` | Filter by question type; empty = all. | +| `dataset.workspace_root` | Per-item workspace root (`benchmark/longmemeval/workspaces/longmemeval-s`). | +| `evaluation.num_workers` | `0` = auto (cpu-2), `1` = sequential, `>1` = parallel. | +| `evaluation.filter_future_sessions` | Only ingest sessions with timestamp ≤ `question_date`. | +| `reme.config` | ReMe config used (`lme.yaml`). | +| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | Dream triggering behavior. | +| `output.dir` | Results directory (`benchmark/longmemeval/results`). | + +## 5. Outputs + +Results are JSON files written to `output.dir` as `results_.json`, +with a per-type accuracy summary also printed to the console. Logging +conventions are shared across benchmarks — see the +[top-level README](../README.md#outputs--logs). + +## 6. Reference Results + +### cleaned-s + +**Basic settings** + +1. Modified auto-memory prompt, auto-dream disabled. +2. All sessions in reme-memory are strictly earlier than the question time. + +**Results** + +agentscope==2.0.4.post1, conda reme env, 32 workers, eval-only (reusing prebuilt memory) +(2026-08-06, 500 items, total 10.0 min) + +| Type | Agentic | input tok/q | output tok/q | total tok/q | tool calls/q | +|---|---|---|---|---|---| +| knowledge-update | 0.910 | 31,581 | 589 | 32,169 | 2.90 | +| multi-session | 0.842 | 52,837 | 1,474 | 54,311 | 4.21 | +| single-session-assistant | 1.000 | 15,596 | 279 | 15,875 | 1.89 | +| single-session-preference | 0.633 | 36,802 | 818 | 37,620 | 3.60 | +| single-session-user | 0.986 | 27,433 | 359 | 27,792 | 2.60 | +| temporal-reasoning | 0.902 | 62,674 | 985 | 63,659 | 4.97 | +| **OVERALL** | **0.894** | **43,448** | **876** | **44,324** | **3.69** | \ No newline at end of file diff --git a/benchmark/longmemeval/README_ZH.md b/benchmark/longmemeval/README_ZH.md new file mode 100644 index 00000000..9c07b175 --- /dev/null +++ b/benchmark/longmemeval/README_ZH.md @@ -0,0 +1,90 @@ +# LongMemEval 评测 + +[English version](./README.md) + +LongMemEval 是一个面向**多轮多会话历史的长期记忆能力**的评测基准。每个条目提供一组按时间 +顺序排列的用户与助手之间的会话，以及一个只能通过推理用户自有记忆才能回答的探测问题。ReMe +将会话摄入按条目隔离的工作区，以 agentic（ReAct）模式回答问题，最后由 LLM-as-judge 打分。 + +题型包括单会话（user / assistant / preference）、多会话推理、知识更新与时间推理等。 + +> 公共设置（依赖、凭据、日志约定）见[总评测说明](../README_ZH.md)。 + +## 1. 获取数据集 + +ReMe 仅使用 **cleaned-S** 版本，数据托管在 HuggingFace： +[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)。 +下载脚本经 hf-mirror.com 镜像源获取，如需更换源请修改 [`download.py`](./download.py) 中的 +`BASE_URL`。 + +```bash +cd benchmark/longmemeval +python download.py # 保存为 dataset/longmemeval_s_reme_cleaned.json，已存在则自动跳过 +``` + +ground truth 已内嵌在数据文件中。 + +## 2. 运行 + +在仓库根目录执行： + +```bash +python benchmark/longmemeval/run.py +python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml +python benchmark/longmemeval/run.py -q # 安静模式：仅评测级日志 +python benchmark/longmemeval/run.py --log-level WARNING # 降低评测 runner 日志 +python benchmark/longmemeval/run.py --reme-log-level WARNING # 降低 reme 内部日志 +python benchmark/longmemeval/run.py --eval_only # 复用已有工作区，仅执行查询 + 评判 +``` + +## 3. 流程 + +1. 加载数据集（ground truth 已内嵌在数据文件中）。 +2. 为每个条目创建独立工作区，按时间顺序摄入会话。 +3. 当相邻会话跨越配置的时刻（默认 23:00）时触发 `auto_dream`。 +4. 以 agentic（ReAct）模式回答每个问题。 +5. 通过 `answer_judge` 任务对答案做二元（yes/no）评判，并输出各类型准确率。 + +## 4. 关键配置 —— `benchmark/longmemeval/config.yaml` + +| 配置项 | 含义 | +| --- | --- | +| `dataset.path` | 待评测的数据集文件（如 `longmemeval_s_reme_cleaned.json`），已包含 ground truth。 | +| `dataset.start_index` / `num_items` | 评测条目的切片范围。 | +| `dataset.question_types` | 按问题类型过滤，空表示全部。 | +| `dataset.workspace_root` | 条目工作区根目录（`benchmark/longmemeval/workspaces/longmemeval-s`）。 | +| `evaluation.num_workers` | `0` = 自动（cpu-2），`1` = 串行，`>1` = 并行。 | +| `evaluation.filter_future_sessions` | 仅摄入时间戳 ≤ `question_date` 的会话。 | +| `reme.config` | 使用的 ReMe 配置（`lme.yaml`）。 | +| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | dream 触发行为。 | +| `output.dir` | 结果目录（`benchmark/longmemeval/results`）。 | + +## 5. 输出 + +结果以 JSON 文件写入 `output.dir`，文件名为 `results_.json`， +同时控制台会打印含各类型准确率的汇总。日志约定在各基准间通用，见 +[总说明](../README_ZH.md#输出与日志)。 + +## 6. 参考结果 + +### cleaned-s + +**基础设置** + +1. 使用修改后的 auto-memory prompt，关闭 auto-dream 机制 +2. reme-memory 中的全部 session 的时间一定早于 question 的时间 + +**结果** + +agentscope==2.0.4.post1, conda reme env, 32 workers, eval-only（复用预构建记忆） +（2026-08-06，500 题，总计 10.0 min） + +| 类型 | Agentic | input tok/q | output tok/q | total tok/q | tool calls/q | +|---|---|---|---|---|---| +| knowledge-update | 0.910 | 31,581 | 589 | 32,169 | 2.90 | +| multi-session | 0.842 | 52,837 | 1,474 | 54,311 | 4.21 | +| single-session-assistant | 1.000 | 15,596 | 279 | 15,875 | 1.89 | +| single-session-preference | 0.633 | 36,802 | 818 | 37,620 | 3.60 | +| single-session-user | 0.986 | 27,433 | 359 | 27,792 | 2.60 | +| temporal-reasoning | 0.902 | 62,674 | 985 | 63,659 | 4.97 | +| **OVERALL** | **0.894** | **43,448** | **876** | **44,324** | **3.69** | diff --git a/benchmark/longmemeval/kill.sh b/benchmark/longmemeval/kill.sh new file mode 100644 index 00000000..9f4a5237 --- /dev/null +++ b/benchmark/longmemeval/kill.sh @@ -0,0 +1,76 @@ +#!/bin/bash +# 杀死指定进程及其所有子进程 +# Usage: bash kill.sh + +if [ -z "$1" ]; then + echo "Usage: bash kill.sh " + echo " 杀死指定进程及其所有子进程" + exit 1 +fi + +PID=$1 + +# 检查进程是否存在 +if ! kill -0 "$PID" 2>/dev/null; then + echo "进程 $PID 不存在" + exit 1 +fi + +# 递归收集所有子进程（包括子进程的子进程） +collect_children() { + local parent=$1 + local children + children=$(ps -o pid= --ppid "$parent" 2>/dev/null | tr -d ' ') + for child in $children; do + collect_children "$child" + done + echo "$parent" +} + +# 收集进程树（子进程在前，父进程在后，保证先杀子再杀父） +PROCESS_TREE=$(collect_children "$PID") +TOTAL=$(echo "$PROCESS_TREE" | wc -l | tr -d ' ') + +echo "进程树（共 $TOTAL 个进程）:" +while read -r p; do + cmd=$(ps -o args= -p "$p" 2>/dev/null | head -c 80) + printf " PID=%-8s %s\n" "$p" "$cmd" +done <<< "$PROCESS_TREE" + +# 先 SIGTERM 优雅终止 +echo "" +echo "发送 SIGTERM..." +while read -r p; do + kill "$p" 2>/dev/null +done <<< "$PROCESS_TREE" + +# 等待最多 5 秒 +for i in $(seq 1 5); do + alive=false + while read -r p; do + if kill -0 "$p" 2>/dev/null; then + alive=true + fi + done <<< "$PROCESS_TREE" + if [ "$alive" = false ]; then + break + fi + sleep 1 +done + +# 检查是否还有残留，强制 SIGKILL +remaining=false +while read -r p; do + if kill -0 "$p" 2>/dev/null; then + remaining=true + fi +done <<< "$PROCESS_TREE" + +if [ "$remaining" = true ]; then + echo "部分进程未响应，发送 SIGKILL..." + while read -r p; do + kill -9 "$p" 2>/dev/null + done <<< "$PROCESS_TREE" +fi + +echo "已终止进程树（根 PID=$PID，共 $TOTAL 个进程）" diff --git a/benchmark/results_md/beam.md b/benchmark/results_md/beam.md deleted file mode 100644 index 651690d7..00000000 --- a/benchmark/results_md/beam.md +++ /dev/null @@ -1,54 +0,0 @@ -# beam result - -## longmemeval版本的prompt - -### 100K - - -agentscope==2.0.4.post1, conda reme 环境, 20 并发, eval-only（复用已构建 memory） -(2026-08-05, 20 cases / 400 Qs, 总耗时 46.0 min) - -| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q | -|---|---|---|---|---|---|---| -| abstention | 0.550 | 0.550 | 96,031 | 1,070 | 97,101 | 4.58 | -| contradiction_resolution | 0.438 | 0.412 | 32,263 | 872 | 33,135 | 2.48 | -| event_ordering | 0.501 | 0.423 | 140,195 | 5,163 | 145,358 | 4.70 | -| information_extraction | 0.873 | 0.832 | 50,245 | 883 | 51,128 | 3.15 | -| instruction_following | 0.750 | 0.725 | 37,986 | 848 | 38,834 | 2.67 | -| knowledge_update | 0.688 | 0.675 | 31,198 | 651 | 31,849 | 2.27 | -| multi_session_reasoning | 0.626 | 0.584 | 85,038 | 4,563 | 89,601 | 4.28 | -| preference_following | 0.925 | 0.912 | 34,281 | 989 | 35,270 | 2.50 | -| summarization | 0.623 | 0.461 | 89,657 | 2,056 | 91,713 | 4.12 | -| temporal_reasoning | 0.637 | 0.625 | 34,563 | 1,049 | 35,612 | 2.52 | -| **OVERALL** | **0.661** | **0.620** | **63,146** | **1,814** | **64,960** | **3.33** | - -Memory Construction 平均 token 消耗（default agent, 20 cases 全量构建）: - -| Agent | input tok/case | output tok/case | total tok/case | -|---|---|---|---| -| default | 2,172,316 | 136,697 | 2,309,013 | - -### 1M - -agentscope==2.0.4.post1, conda reme 环境, 20 并发, 全量构建 memory -(2026-08-05, 35 cases / 700 Qs, 总耗时 459.2 min) - -| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q | -|---|---|---|---|---|---|---| -| abstention | 0.429 | 0.429 | 118,707 | 1,178 | 119,886 | 4.20 | -| contradiction_resolution | 0.391 | 0.364 | 49,787 | 810 | 50,597 | 2.50 | -| event_ordering | 0.558 | 0.456 | 201,514 | 3,889 | 205,403 | 4.79 | -| information_extraction | 0.809 | 0.772 | 78,950 | 894 | 79,844 | 3.00 | -| instruction_following | 0.852 | 0.832 | 55,757 | 924 | 56,681 | 2.81 | -| knowledge_update | 0.779 | 0.771 | 45,981 | 665 | 46,646 | 2.37 | -| multi_session_reasoning | 0.658 | 0.612 | 138,133 | 2,873 | 141,006 | 4.40 | -| preference_following | 0.798 | 0.777 | 51,796 | 920 | 52,716 | 2.53 | -| summarization | 0.693 | 0.537 | 158,794 | 2,905 | 161,700 | 4.44 | -| temporal_reasoning | 0.536 | 0.536 | 100,176 | 3,148 | 103,324 | 3.90 | -| **OVERALL** | **0.650** | **0.609** | **99,959** | **1,821** | **101,780** | **3.49** | - -Memory Construction 平均 token 消耗（default agent, 35 cases 全量构建）: - -| Agent | input tok/case | output tok/case | total tok/case | -|---|---|---|---| -| default | 31,943,817 | 1,417,061 | 33,360,878 | \ No newline at end of file diff --git a/benchmark/results_md/longmemeval.md b/benchmark/results_md/longmemeval.md deleted file mode 100644 index 59671395..00000000 --- a/benchmark/results_md/longmemeval.md +++ /dev/null @@ -1,92 +0,0 @@ -# LongMemEval 数据集测试结果 - -## cleaned-s - -**basic settings** - -1. 使用修改后的auto-memory prompt，关闭auto-dream机制 -2. reme-memory中的全部session的时间一定早于question的时间 - -**results ** - -1. Agentic answer框架回答，每次最多调用5次search - -| Category | Total | Correct | Wrong | Accuracy | -|---|---|---|---|---| -| single-session-user | 70 | 66 | 4 | 94.3% | -| single-session-assistant | 56 | 52 | 4 | 92.9% | -| knowledge-update | 78 | 60 | 18 | 76.9% | -| multi-session | 133 | 93 | 40 | 69.9% | -| temporal-reasoning | 133 | 78 | 55 | 58.6% | -| single-session-preference | 30 | 8 | 22 | 26.7% | -| **Overall** | **500** | **357** | **143** | **71.4%** | - -2. prompted-based amswer，每次固定使用原始query召回10个fileChunk - -| Category | Total | Correct | Wrong | Accuracy | -|---|---|---|---|---| -| single-session-assistant | 56 | 56 | 0 | 100.0% | -| single-session-user | 70 | 67 | 3 | 95.7% | -| knowledge-update | 78 | 69 | 9 | 88.5% | -| multi-session | 133 | 99 | 34 | 74.4% | -| temporal-reasoning | 133 | 83 | 50 | 62.4% | -| single-session-preference | 30 | 16 | 14 | 53.3% | -| **Overall** | **500** | **390** | **110** | **78.0%** | - -3. golden session。 使用与prompt-based answer相似的方法，唯一区别是，输入的chunk是longMemEval提供的golden session。 - -| Category | Total | Correct | Wrong | Accuracy | -|---|---|---|---|---| -| single-session-assistant | 56 | 56 | 0 | 100.0% | -| single-session-user | 70 | 69 | 1 | 98.6% | -| knowledge-update | 78 | 74 | 4 | 94.9% | -| temporal-reasoning | 133 | 124 | 9 | 93.2% | -| multi-session | 133 | 117 | 16 | 88.0% | -| single-session-preference | 30 | 17 | 13 | 56.7% | -| **Overall** | **500** | **457** | **43** | **91.4%** | - -4. golden session + time filter. 和上面一个实验的区别是，输入的golden被过滤了一次，要求输入session的时间戳必须早于question的时间才行。 - -一共被过滤掉了75个session，44个question受到了影响。temperal reasoning类型受影响最大。有20个case不包含任何一个groundtruth session。 根据golden session回答正确并且golden session非空，一共有424个case。 - -| Category | Total | Correct | Wrong | Accuracy | -|---|---|---|---|---| -| knowledge-update | 78 | 75 | 3 | 96.2% | -| single-session-user | 70 | 67 | 3 | 95.7% | -| multi-session | 133 | 122 | 11 | 91.7% | -| single-session-assistant | 56 | 55 | 1 | 98.2% | -| temporal-reasoning | 133 | 91 | 42 | 68.4% | -| single-session-preference | 30 | 16 | 14 | 53.3% | -| **Overall** | **500** | **426** | **74** | **85.2%** | - -5. 关闭auto-memory机制，根据原始query一次性混合检索召回原始session，计算recall. - -| Category | Total | yes-judge | recall@5 / yes | recall@10 / yes | -|---|---|---|---|---| -| knowledge-update | 78 | 75 | 99.3% | 100% | -| single-session-user | 70 | 67 | 100% | 100% | -| multi-session | 133 | 122 | 91.8% | 95.8% | -| single-session-assistant | 56 | 55 | 100% | 100% | -| temporal-reasoning | 133 | 91 | 87.6% | 94.2% | -| single-session-preference | 30 | 16 | 100% | 100% | -| **Overall** | **500** | **426** | **87.6%** | **94.2%** | - - -## 最终groundtruth - -### agentic + prompted（最终GT，2026-07-16） - - -| Category | Total | Agentic | Prompted limit=15 | -|---|---|---|---| -| single-session-assistant | 56 | 56/56 (100.0%) | 54/56 (96.4%) | -| single-session-user | 70 | 66/70 (94.3%) | 62/70 (88.6%) | -| knowledge-update | 78 | 75/78 (96.2%) | 67/78 (85.9%) | -| temporal-reasoning | 133 | 122/133 (91.7%) | 117/133 (88.0%) | -| multi-session | 133 | 115/133 (86.5%) | 101/133 (75.9%) | -| single-session-preference | 30 | 21/30 (70.0%) | 10/30 (33.3%) | -| **Overall** | **500** | **455/500 (91.0%)** | **411/500 (82.2%)** | - -Prompted token 消耗：总 input 13,111,421 (平均 26,275/题)，总 output 313,370 (平均 628/题)。 -平均 sessions_ingested: 44.8，dreams_triggered: 0。 - diff --git a/reme/steps/benchmark/lme/agentic_answer.yaml b/reme/steps/benchmark/lme/agentic_answer.yaml index 9d125086..153d211b 100644 --- a/reme/steps/benchmark/lme/agentic_answer.yaml +++ b/reme/steps/benchmark/lme/agentic_answer.yaml @@ -1,25 +1,16 @@ system_prompt: | - You are a memory retrieval assistant. A memory system manages the user's conversation history. Given a query, you MUST call the `search` tool at least once to retrieve relevant content from the memory system before answering. - - ## Memory System - The memory system stores the user's history in three forms: - - **Summary notes**: condensed knowledge distilled from past sessions. - - **Compressed session chunks**: fragments of session transcripts compressed to drop low-information phrasing; the dialogue structure is generally kept, but some detail may be shortened or lost. - - **Original sessions**: complete verbatim transcripts of past conversations, stored as memory files. - - The `search` tool retrieves summary notes and compressed session chunks; it does NOT return original sessions directly. Every result is a chunk — a fragment of a memory file identified by its path — and may lack surrounding context. Compressed session chunks are marked with a leading `compressed session chunk:` notice. When a chunk looks highly relevant to the query but its content is vague, truncated, or missing detail due to compression, use the `read` tool with the chunk's path to read the original session before drawing conclusions. - - ## Workflow - 1. Use `search` to locate possibly relevant content. It is a hybrid retrieval tool combining BM25 keyword matching and vector semantic search, ensuring diverse recall. Call it multiple times with different query phrasings to gather comprehensive information; do not stop after the first relevant result unless you are confident the information is sufficient. - 2. When a retrieved chunk is strongly related to the query but ambiguous because of compression, call `read` on its path to recover the full original session, and ground your answer on that. - 3. Use `add_draft` to save key findings during the search, and `read_all_draft` to review all saved notes before answering. - 4. After finding a preliminary answer, continue searching from multiple angles to check for potentially conflicting, supplementary, or updated information. The goal is to improve the accuracy and completeness of the final answer. - 5. Stop searching and generate the final answer once you are confident that further searches will not yield additional useful information. - - Your total tool calls should be at most 9 times. + You are a memory retrieval assistant. You MUST use the search tool to find information before answering. + - Your total time of tool calls should be at most 9 times + ## Search Strategy + - You can call 'search' tool to search multiple times (at least once) with different queries to gather comprehensive information. + - Do not stop after the first relevant one unless you are confident that the information is sufficient. + - You can use 'read' tool to read specific parts of a specific file. + - After finding a preliminary answer, continue searching from multiple angles to check for potentially conflicting, supplementary, or updated information. + ## Draft Tool + - Use 'add_draft' to save key findings during search, and 'read_all_draft' to review all saved notes before answering. ## Answer Rules - - Answer based ONLY on retrieved or read context. - - If memories contain contradictory information, weigh the context, query, and most importantly the timestamp of each memory to determine the final answer. + - Answer based ONLY on retrieved context. - Output ONLY the direct factual answer — no reasoning, no search process, no elaboration. - Do NOT invent or assume information that isn't in the memories. Do NOT extrapolate or fabricate connections from related but non-direct evidence. - If information is not found or not sufficient after multiple searches, or if the maximum iteration count is reached without finding answer, reply: 'Information not found.'