mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-08-28 05:25:04 +00:00
refactor(benchmark): isolate per-benchmark assets and simplify LME agentic prompt (#422)
* chore(benchmark): isolate dataset/workspaces/results per benchmark
- Move shared benchmark/{datasets,memory_workspaces,results} into per-benchmark subdirs benchmark/<name>/{dataset,workspaces,results}
- Update beam/longmemeval config.yaml and run.py path defaults
- Relocate longmemeval download.py to benchmark/longmemeval/ (downloads into dataset/ subdir); inline dataset download docs into README
- Update .gitignore: benchmark/*/{dataset,workspaces,results}/
- Move result-{beam,longmemeval}.md to benchmark/results_md/ and drop result- prefix; update README links
- Fix stale path refs in llm_judge.py and logs/demo_search_format.py
* feat(benchmark): add read tool to agentic answer and update BEAM results
- Add 'read' to job_tools in BaseAgenticAnswerStep for file reading capability
- Document read tool usage in lme/agentic_answer.yaml system prompt
- Update result-beam.md with latest evaluation scores (OVERALL: 0.623/0.580)
* feat(auto_memory): add source line-number markers for note traceability
- Add _format_history hook in AutoMemoryStep with line-number annotation
- Override in BeamAutoMemoryStep to prefix each turn with [Ln] for citation
- Add session_file variable to prompt templates for source marker paths
- Simplify repeated extraction rules by referencing system prompt
- Enhance agentic_answer search strategy (multi-search, read tool hint)
- Add warning log on ReadStep failure
* feat(beam): enhance auto_memory with source markers and pilot ingest tooling
* refactor(beam): rename max_chunk_words to max_segment_words, drop one-off pilot scripts
* feat: add CompressorStep and search_v2 dual-mode session compression
- Add CompressorStep (reme/steps/evolve/compressor.py) for direct LLM
text compression with optional query-guided relevance filtering
- Extend search_v2_step to support query-aware and query-independent
session transcript compression via _compress injected kwargs
- Refactor _source_format.py: split into render_chunk_entries +
join_chunk_entries; session chunks now render line-aligned with
L<n>: prefixes for verbatim/compressed parity
- Add JOB_TOOLS and INJECTED_JOB_KWARGS to BaseAgenticAnswerStep for
per-subclass tool and parameter injection
- LmeAgenticAnswerStep injects _search._compress payload to enable
query-aware compression during benchmark evaluation
- Record compression ablation results in result-longmemeval.md
- Add unit tests for CompressorStep and search compression paths
* refactor(compress): relax session compression to lenient format-preserving strategy and update LME results
* refactor(benchmark): make session compression config-driven via compress_session flag
Move session-transcript compression from LME hard-coded injection to a
runtime context flag set by evaluation.compress_session in each
benchmark config. Compression is off by default for both BEAM and LME,
and BaseAgenticAnswerStep now conditionally injects the _search compress
payload only when the flag is truthy.
* feat(lme/auto_memory): add source attribution markers with line numbers
Add _format_history to annotate each turn with [Ln] line numbers and
expose {session_file} in prompts so the agent can emit bare wikilink-style
source markers like [[session/dialog/s1.jsonl#L1-L2,L5-L6]] at the end
of factual entries. Consolidate the per-prompt body/format rules into
references to the system prompt to avoid drift, and add frontmatter-
protection guidance for the edit tool.
* feat: improve agentic answer prompt and update beam 100K results
- Strengthen abstention rule: prohibit extrapolation from related but
non-direct evidence
- Add multi-angle search after preliminary answer to check for
conflicting/supplementary/updated information
- Add max-iteration fallback to 'Information not found'
- Update beam.md with 100K results (agentscope 2.0.4.post1, from scratch)
including per-type token consumption and memory construction stats
- config.yaml: 100K dataset, 20 workers for BEAM evaluation
- run.py: add memory construction token usage tracking (default agent)
- Overall: 0.635 → 0.654 (+0.019), contradiction_resolution: 0.338 → 0.478
(+0.140), abstention: 0.500 → 0.525 (+0.025)
* feat(read): add session-aware formatting for read tool and update BEAM eval
- Add truncate_session_output in _file_io.py to render jsonl session
lines as [speaker @ time] content before byte-budget truncation
- Add read_step_format_session flag to ReadStep, honoring injected
job kwargs (precedence) and YAML fallback
- Inject read_step_format_session=True into BaseAgenticAnswerStep
so agentic answer reads render session transcripts human-readably
- Refine BEAM agentic_answer prompt: continue multi-angle search
after preliminary answer, forbid fabrication/extrapolation
- Update BEAM config to 1M variant and add sequential 100K-eval /
1M-build shell script
- Refresh benchmark/results_md/beam.md with latest results
* chore(config): disable expand_links in beam and lme search_v2 configs
* refactor(beam): drop one-off sequential 100K-eval-then-1M-build script
* fix(benchmark): add compressor job to beam config and fix BEAM clone instructions
- Add compressor job and compressor as_llm component to reme/config/beam.yaml
(aligned with lme.yaml) so that compress_session: true works for BEAM
- Add graceful degradation guard in search_v2._compress_session_entries:
when the compressor job is missing from the active config, log a warning
and skip compression instead of raising 'Job compressor not found'.
Skipped when there is no app_context so unit tests mocking run_job still
drive compression behavior.
- Fix BEAM download instructions in README.md/README_ZH.md: add mkdir -p
before cd benchmark/beam/dataset (the directory is gitignored and absent
in a fresh clone)
* fix(steps): guard compressor exceptions and fix ReadStep boolean override
1. search_v2: catch per-entry exceptions from run_job('compressor') inside
compress() so asyncio.gather never propagates a compressor failure (e.g.
temporary LLM outage). The failing entry keeps its original body while
remaining entries are still compressed, preserving already-retrieved
search results.
2. read: replace 'context_value or yaml_value' with an existence check so
that a runtime-injected False can explicitly disable a YAML-true
read_step_format_session flag.
Add focused unit tests for both paths.
* fix(search_v2): use existence check for strict_date_filter boolean override
Replace 'context_value or yaml_value' with an existence-based check so
that a runtime-injected False can explicitly disable a YAML-true
strict_date_filter flag, consistent with the read_step_format_session fix.
* refactor(search): simplify strict_date_filter fallback to truthiness-or
* style(test): rename unused param to satisfy pylint W0613
* refactor(benchmark): isolate per-benchmark assets and simplify LME agentic prompt
- Move shared benchmark/README, README_ZH, kill.sh, and results_md/*.md into
per-benchmark subdirs (benchmark/beam/, benchmark/longmemeval/) so each
benchmark owns its own docs, scripts, and result snapshots.
- Simplify lme/agentic_answer.yaml system prompt: drop verbose memory-system
description, keep search strategy, draft tool, and answer rules concise.
* docs(benchmark): update LME README_ZH results to latest eval run
---------
Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
This commit is contained in:
xyf2020
GitHub
parent
f31daf1949
commit
23d4c96c15
11 changed files with 515 additions and 469 deletions
|
|
@ -1,155 +0,0 @@
|
|||
[中文版 / Chinese version](./README_ZH.md)
|
||||
|
||||
# ReMe Benchmarks
|
||||
|
||||
Reproduction guide for the two memory benchmarks shipped with ReMe:
|
||||
|
||||
- **LongMemEval** — long-term memory over multi-session chat histories.
|
||||
- **BEAM** — memory capability over long-context chat cases with rubric-based judging.
|
||||
|
||||
Each benchmark runs its own end-to-end pipeline: ingest sessions into an isolated
|
||||
per-item workspace, answer probing questions via an agentic (ReAct) mode,
|
||||
then score answers with an LLM-as-judge.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
Install ReMe with dev + core extras (Python 3.11+):
|
||||
|
||||
```bash
|
||||
pip install -e ".[dev,core]"
|
||||
```
|
||||
|
||||
Configure model credentials in a project-root `.env` file (copied from `example.env`).
|
||||
The runners auto-load `.env` from the repository root. Required variables typically include:
|
||||
|
||||
```bash
|
||||
LLM_API_KEY=...
|
||||
LLM_BASE_URL=...
|
||||
EMBEDDING_API_KEY=...
|
||||
EMBEDDING_BASE_URL=...
|
||||
```
|
||||
|
||||
Model names and component wiring live in the ReMe configs referenced by each benchmark
|
||||
(`reme/config/lme.yaml` and `reme/config/beam.yaml`).
|
||||
|
||||
## 2. Download Datasets
|
||||
|
||||
Each benchmark keeps its own data under its directory:
|
||||
`benchmark/<name>/dataset` (input data), `benchmark/<name>/workspaces`
|
||||
(per-item memory workspaces), and `benchmark/<name>/results` (evaluation outputs).
|
||||
All three are excluded from Git.
|
||||
|
||||
**LongMemEval** — ReMe uses only the **cleaned-S** split, hosted on HuggingFace:
|
||||
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)
|
||||
(the script downloads via the hf-mirror.com mirror; to use a different mirror,
|
||||
modify `BASE_URL` in `download.py`):
|
||||
|
||||
```bash
|
||||
cd benchmark/longmemeval
|
||||
python download.py # saves dataset/longmemeval_s_reme_cleaned.json; skips if already present
|
||||
```
|
||||
|
||||
**BEAM** (public repository, cloned into `benchmark/beam/dataset/`):
|
||||
|
||||
```bash
|
||||
mkdir -p benchmark/beam/dataset
|
||||
cd benchmark/beam/dataset
|
||||
git clone https://github.com/mohammadtavakoli78/BEAM.git
|
||||
```
|
||||
|
||||
After cloning, `benchmark/beam/dataset/BEAM/` should contain `chats/`, `src/`,
|
||||
`topics/` and other subdirectories.
|
||||
|
||||
## 3. Run LongMemEval
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
python benchmark/longmemeval/run.py
|
||||
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
|
||||
python benchmark/longmemeval/run.py -q # quiet: only eval-level logs
|
||||
python benchmark/longmemeval/run.py --log-level WARNING # reduce eval runner logs
|
||||
python benchmark/longmemeval/run.py --reme-log-level WARNING # reduce reme internal logs
|
||||
python benchmark/longmemeval/run.py --eval_only # reuse existing workspaces, query + judge only
|
||||
```
|
||||
|
||||
### Pipeline
|
||||
|
||||
1. Load the dataset (ground truth is embedded in the data file).
|
||||
2. For each item, create an isolated workspace and ingest sessions in chronological order.
|
||||
3. Trigger `auto_dream` when consecutive sessions cross the configured hour (default 23:00).
|
||||
4. Answer each question via agentic (ReAct) mode.
|
||||
5. Judge the answer (binary yes/no) with the `answer_judge` job and print per-type accuracy.
|
||||
|
||||
### Key config — `benchmark/longmemeval/config.yaml`
|
||||
|
||||
| Key | Meaning |
|
||||
| --- | --- |
|
||||
| `dataset.path` | Dataset file to evaluate (e.g. `longmemeval_s_reme_cleaned.json`); ground truth is included. |
|
||||
| `dataset.start_index` / `num_items` | Slice of items to evaluate. |
|
||||
| `dataset.question_types` | Filter by question type; empty = all. |
|
||||
| `dataset.workspace_root` | Per-item workspace root (`benchmark/longmemeval/workspaces/longmemeval-s`). |
|
||||
| `evaluation.num_workers` | `0` = auto (cpu-2), `1` = sequential, `>1` = parallel. |
|
||||
| `evaluation.filter_future_sessions` | Only ingest sessions with timestamp ≤ `question_date`. |
|
||||
| `reme.config` | ReMe config used (`lme.yaml`). |
|
||||
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | Dream triggering behavior. |
|
||||
| `output.dir` | Results directory (`benchmark/longmemeval/results`). |
|
||||
|
||||
## 4. Run BEAM
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
python benchmark/beam/run.py
|
||||
python benchmark/beam/run.py --config benchmark/beam/config.yaml
|
||||
python benchmark/beam/run.py -q # quiet
|
||||
python benchmark/beam/run.py --eval_only # reuse existing workspaces, query + judge only
|
||||
```
|
||||
|
||||
### Pipeline
|
||||
|
||||
1. For each case, load `chat.json` and convert each batch into a ReMe session.
|
||||
2. Ingest sessions in chronological order into an isolated workspace, then `digest_update`.
|
||||
3. Answer each probing question via agentic (ReAct) mode.
|
||||
4. Score answers with BEAM's rubric-based `answer_judge` job and print per-type averages.
|
||||
|
||||
### Key config — `benchmark/beam/config.yaml`
|
||||
|
||||
| Key | Meaning |
|
||||
| --- | --- |
|
||||
| `dataset.beam_root` | BEAM dataset root (`benchmark/beam/dataset/BEAM`). |
|
||||
| `dataset.chat_size` | Variant to run: `100K` / `500K` / `1M` / `10M`. |
|
||||
| `dataset.case_ids` | Specific cases (e.g. `["1","2"]`); empty = all cases. |
|
||||
| `dataset.start_index` / `num_items` | Case pagination (`num_items` `0` = all). |
|
||||
| `dataset.workspace_root` | Per-case workspace root (`benchmark/beam/workspaces/beam`). |
|
||||
| `evaluation.num_workers` | `0` = auto, `1` = sequential, `>1` = parallel. |
|
||||
| `reme.config` | ReMe config used (`beam.yaml`). |
|
||||
| `output.dir` | Results directory (`benchmark/beam/results`). |
|
||||
|
||||
## 5. Outputs & Logs
|
||||
|
||||
- **Results**: JSON files written to `output.dir`
|
||||
(`results_<timestamp>.json` for LongMemEval,
|
||||
`results_<chat_size>_<timestamp>.json` for BEAM). A summary with per-type
|
||||
accuracy/score is also printed to the console.
|
||||
- **Logs**: when `output.log_to_file` is enabled, per-run logs are written to
|
||||
`logs/<log_prefix>_<timestamp>/` (a `runner.log` plus one `worker-<pid>.log`
|
||||
per worker process).
|
||||
|
||||
## 6. Stopping a Run
|
||||
|
||||
Parallel runs spawn a process tree. To terminate a run and all its workers cleanly:
|
||||
|
||||
```bash
|
||||
bash benchmark/kill.sh <PID>
|
||||
```
|
||||
|
||||
The script gracefully sends `SIGTERM` to the whole process tree, then escalates to
|
||||
`SIGKILL` for any process that does not exit within 5 seconds.
|
||||
|
||||
## 7. Reference Results
|
||||
|
||||
Recorded evaluation results are available in:
|
||||
|
||||
- [`longmemeval.md`](./results_md/longmemeval.md)
|
||||
- [`beam.md`](./results_md/beam.md)
|
||||
|
|
@ -1,149 +0,0 @@
|
|||
# ReMe 评测复现说明
|
||||
|
||||
ReMe 内置两个记忆能力评测基准的复现指南：
|
||||
|
||||
- **LongMemEval** —— 面向多轮多会话历史的长期记忆能力评测。
|
||||
- **BEAM** —— 面向长上下文对话场景、基于评分细则（rubric）打分的记忆能力评测。
|
||||
|
||||
每个基准都会运行完整的端到端流程：将会话摄入独立的按条目隔离的工作区，
|
||||
以 agentic（ReAct）模式回答探测问题，
|
||||
最后由 LLM-as-judge 对答案进行打分。
|
||||
|
||||
## 1. 环境准备
|
||||
|
||||
安装 ReMe 及 dev + core 附加依赖（Python 3.11+）：
|
||||
|
||||
```bash
|
||||
pip install -e ".[dev,core]"
|
||||
```
|
||||
|
||||
在项目根目录配置 `.env`（可从 `example.env` 复制），运行脚本会自动从仓库根目录加载 `.env`。
|
||||
通常需要以下变量：
|
||||
|
||||
```bash
|
||||
LLM_API_KEY=...
|
||||
LLM_BASE_URL=...
|
||||
EMBEDDING_API_KEY=...
|
||||
EMBEDDING_BASE_URL=...
|
||||
```
|
||||
|
||||
模型名称与组件装配位于各基准引用的 ReMe 配置中
|
||||
（`reme/config/lme.yaml` 与 `reme/config/beam.yaml`）。
|
||||
|
||||
## 2. 下载数据集
|
||||
|
||||
每个基准的数据都存放在各自目录下：`benchmark/<name>/dataset`（输入数据）、
|
||||
`benchmark/<name>/workspaces`（按条目隔离的记忆工作区）、
|
||||
`benchmark/<name>/results`（评测输出）。三者均不纳入 Git 版本管理。
|
||||
|
||||
**LongMemEval** —— ReMe 仅使用 **cleaned-S** 版本，数据托管在 HuggingFace：
|
||||
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)
|
||||
（下载脚本经 hf-mirror.com 镜像源获取，如需更换源请修改 `download.py` 中的 `BASE_URL`）：
|
||||
|
||||
```bash
|
||||
cd benchmark/longmemeval
|
||||
python download.py # 保存为 dataset/longmemeval_s_reme_cleaned.json，已存在则自动跳过
|
||||
```
|
||||
|
||||
**BEAM**（公开仓库，clone 到 `benchmark/beam/dataset/` 下）：
|
||||
|
||||
```bash
|
||||
mkdir -p benchmark/beam/dataset
|
||||
cd benchmark/beam/dataset
|
||||
git clone https://github.com/mohammadtavakoli78/BEAM.git
|
||||
```
|
||||
|
||||
clone 完成后，`benchmark/beam/dataset/BEAM/` 目录下应包含 `chats/`、`src/`、`topics/` 等子目录。
|
||||
|
||||
## 3. 运行 LongMemEval
|
||||
|
||||
在仓库根目录执行：
|
||||
|
||||
```bash
|
||||
python benchmark/longmemeval/run.py
|
||||
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
|
||||
python benchmark/longmemeval/run.py -q # 安静模式：仅评测级日志
|
||||
python benchmark/longmemeval/run.py --log-level WARNING # 降低评测 runner 日志
|
||||
python benchmark/longmemeval/run.py --reme-log-level WARNING # 降低 reme 内部日志
|
||||
python benchmark/longmemeval/run.py --eval_only # 复用已有工作区，仅执行查询 + 评判
|
||||
```
|
||||
|
||||
### 流程
|
||||
|
||||
1. 加载数据集（ground truth 已内嵌在数据文件中）。
|
||||
2. 为每个条目创建独立工作区，按时间顺序摄入会话。
|
||||
3. 当相邻会话跨越配置的时刻（默认 23:00）时触发 `auto_dream`。
|
||||
4. 以 agentic（ReAct）模式回答每个问题。
|
||||
5. 通过 `answer_judge` 任务对答案做二元（yes/no）评判，并输出各类型准确率。
|
||||
|
||||
### 关键配置 —— `benchmark/longmemeval/config.yaml`
|
||||
|
||||
| 配置项 | 含义 |
|
||||
| --- | --- |
|
||||
| `dataset.path` | 待评测的数据集文件（如 `longmemeval_s_reme_cleaned.json`），已包含 ground truth。 |
|
||||
| `dataset.start_index` / `num_items` | 评测条目的切片范围。 |
|
||||
| `dataset.question_types` | 按问题类型过滤，空表示全部。 |
|
||||
| `dataset.workspace_root` | 条目工作区根目录（`benchmark/longmemeval/workspaces/longmemeval-s`）。 |
|
||||
| `evaluation.num_workers` | `0` = 自动（cpu-2），`1` = 串行，`>1` = 并行。 |
|
||||
| `evaluation.filter_future_sessions` | 仅摄入时间戳 ≤ `question_date` 的会话。 |
|
||||
| `reme.config` | 使用的 ReMe 配置（`lme.yaml`）。 |
|
||||
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | dream 触发行为。 |
|
||||
| `output.dir` | 结果目录（`benchmark/longmemeval/results`）。 |
|
||||
|
||||
## 4. 运行 BEAM
|
||||
|
||||
在仓库根目录执行：
|
||||
|
||||
```bash
|
||||
python benchmark/beam/run.py
|
||||
python benchmark/beam/run.py --config benchmark/beam/config.yaml
|
||||
python benchmark/beam/run.py -q # 安静模式
|
||||
python benchmark/beam/run.py --eval_only # 复用已有工作区，仅执行查询 + 评判
|
||||
```
|
||||
|
||||
### 流程
|
||||
|
||||
1. 为每个 case 加载 `chat.json`，将每个 batch 转换为一个 ReMe 会话。
|
||||
2. 按时间顺序将会话摄入独立工作区，随后执行 `digest_update`。
|
||||
3. 以 agentic（ReAct）模式回答每个探测问题。
|
||||
4. 通过 BEAM 基于 rubric 的 `answer_judge` 任务打分，并输出各类型平均分。
|
||||
|
||||
### 关键配置 —— `benchmark/beam/config.yaml`
|
||||
|
||||
| 配置项 | 含义 |
|
||||
| --- | --- |
|
||||
| `dataset.beam_root` | BEAM 数据集根目录（`benchmark/beam/dataset/BEAM`）。 |
|
||||
| `dataset.chat_size` | 运行的变体：`100K` / `500K` / `1M` / `10M`。 |
|
||||
| `dataset.case_ids` | 指定 case（如 `["1","2"]`），空表示全部。 |
|
||||
| `dataset.start_index` / `num_items` | case 分页（`num_items` 为 `0` 表示全部）。 |
|
||||
| `dataset.workspace_root` | case 工作区根目录（`benchmark/beam/workspaces/beam`）。 |
|
||||
| `evaluation.num_workers` | `0` = 自动，`1` = 串行，`>1` = 并行。 |
|
||||
| `reme.config` | 使用的 ReMe 配置（`beam.yaml`）。 |
|
||||
| `output.dir` | 结果目录（`benchmark/beam/results`）。 |
|
||||
|
||||
## 5. 输出与日志
|
||||
|
||||
- **结果**：JSON 文件写入 `output.dir`
|
||||
（LongMemEval 为 `results_<timestamp>.json`，
|
||||
BEAM 为 `results_<chat_size>_<timestamp>.json`）。同时控制台会打印含各类型
|
||||
准确率/分数的汇总。
|
||||
- **日志**：当 `output.log_to_file` 开启时，每次运行的日志写入
|
||||
`logs/<log_prefix>_<timestamp>/`（包含一个 `runner.log` 及每个 worker 进程的
|
||||
`worker-<pid>.log`）。
|
||||
|
||||
## 6. 终止运行
|
||||
|
||||
并行运行会派生进程树。若要干净地终止某次运行及其全部 worker：
|
||||
|
||||
```bash
|
||||
bash benchmark/kill.sh <PID>
|
||||
```
|
||||
|
||||
该脚本会先向整个进程树发送 `SIGTERM` 优雅终止，对 5 秒内未退出的进程再升级为 `SIGKILL`。
|
||||
|
||||
## 7. 参考结果
|
||||
|
||||
已记录的评测结果见：
|
||||
|
||||
- [`longmemeval.md`](./results_md/longmemeval.md)
|
||||
- [`beam.md`](./results_md/beam.md)
|
||||
124
benchmark/beam/README.md
Normal file
124
benchmark/beam/README.md
Normal file
|
|
@ -0,0 +1,124 @@
|
|||
[中文版 / Chinese version](./README_ZH.md)
|
||||
|
||||
# BEAM Benchmark
|
||||
|
||||
BEAM is a benchmark for **memory capability over long-context chat cases**. Each
|
||||
case contains a very long chat history split into batches; ReMe converts each
|
||||
batch into a session, ingests them in chronological order, then answers probing
|
||||
questions via an agentic (ReAct) mode. Answers are scored with BEAM's
|
||||
rubric-based `answer_judge` job, which produces both a graded score and a binary
|
||||
verdict, and per-type averages are reported.
|
||||
|
||||
BEAM ships dataset variants by chat size — `100K` / `500K` / `1M` / `10M` — so
|
||||
memory systems can be stressed at different context lengths. Question types
|
||||
include abstention, contradiction resolution, event ordering, information
|
||||
extraction, instruction following, knowledge update, multi-session reasoning,
|
||||
preference following, summarization, and temporal reasoning.
|
||||
|
||||
> For the shared setup (dependencies, credentials, log conventions) see the
|
||||
> [top-level benchmark README](../README.md).
|
||||
|
||||
## 1. Get the Dataset
|
||||
|
||||
BEAM is a public repository, cloned into `benchmark/beam/dataset/`:
|
||||
|
||||
```bash
|
||||
mkdir -p benchmark/beam/dataset
|
||||
cd benchmark/beam/dataset
|
||||
git clone https://github.com/mohammadtavakoli78/BEAM.git
|
||||
```
|
||||
|
||||
After cloning, `benchmark/beam/dataset/BEAM/` should contain `chats/`, `src/`,
|
||||
`topics/` and other subdirectories.
|
||||
|
||||
## 2. Run
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
python benchmark/beam/run.py
|
||||
python benchmark/beam/run.py --config benchmark/beam/config.yaml
|
||||
python benchmark/beam/run.py -q # quiet
|
||||
python benchmark/beam/run.py --eval_only # reuse existing workspaces, query + judge only
|
||||
```
|
||||
|
||||
## 3. Pipeline
|
||||
|
||||
1. For each case, load `chat.json` and convert each batch into a ReMe session.
|
||||
2. Ingest sessions in chronological order into an isolated workspace, then `digest_update`.
|
||||
3. Answer each probing question via agentic (ReAct) mode.
|
||||
4. Score answers with BEAM's rubric-based `answer_judge` job and print per-type averages.
|
||||
|
||||
## 4. Key config — `benchmark/beam/config.yaml`
|
||||
|
||||
| Key | Meaning |
|
||||
| --- | --- |
|
||||
| `dataset.beam_root` | BEAM dataset root (`benchmark/beam/dataset/BEAM`). |
|
||||
| `dataset.chat_size` | Variant to run: `100K` / `500K` / `1M` / `10M`. |
|
||||
| `dataset.case_ids` | Specific cases (e.g. `["1","2"]`); empty = all cases. |
|
||||
| `dataset.start_index` / `num_items` | Case pagination (`num_items` `0` = all). |
|
||||
| `dataset.workspace_root` | Per-case workspace root (`benchmark/beam/workspaces/beam`). |
|
||||
| `evaluation.num_workers` | `0` = auto, `1` = sequential, `>1` = parallel. |
|
||||
| `reme.config` | ReMe config used (`beam.yaml`). |
|
||||
| `output.dir` | Results directory (`benchmark/beam/results`). |
|
||||
|
||||
## 5. Outputs
|
||||
|
||||
Results are JSON files written to `output.dir` as
|
||||
`results_<chat_size>_<timestamp>.json`, with a per-type score summary also
|
||||
printed to the console. Logging conventions are shared across benchmarks — see
|
||||
the [top-level README](../README.md#outputs--logs).
|
||||
|
||||
## 6. Reference Results
|
||||
|
||||
> The results below use the longmemeval-version prompt.
|
||||
|
||||
### 100K
|
||||
|
||||
agentscope==2.0.4.post1, conda reme env, 20 workers, eval-only (reusing prebuilt memory)
|
||||
(2026-08-05, 20 cases / 400 Qs, total 46.0 min)
|
||||
|
||||
| Type | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|
||||
|---|---|---|---|---|---|---|
|
||||
| abstention | 0.550 | 0.550 | 96,031 | 1,070 | 97,101 | 4.58 |
|
||||
| contradiction_resolution | 0.438 | 0.412 | 32,263 | 872 | 33,135 | 2.48 |
|
||||
| event_ordering | 0.501 | 0.423 | 140,195 | 5,163 | 145,358 | 4.70 |
|
||||
| information_extraction | 0.873 | 0.832 | 50,245 | 883 | 51,128 | 3.15 |
|
||||
| instruction_following | 0.750 | 0.725 | 37,986 | 848 | 38,834 | 2.67 |
|
||||
| knowledge_update | 0.688 | 0.675 | 31,198 | 651 | 31,849 | 2.27 |
|
||||
| multi_session_reasoning | 0.626 | 0.584 | 85,038 | 4,563 | 89,601 | 4.28 |
|
||||
| preference_following | 0.925 | 0.912 | 34,281 | 989 | 35,270 | 2.50 |
|
||||
| summarization | 0.623 | 0.461 | 89,657 | 2,056 | 91,713 | 4.12 |
|
||||
| temporal_reasoning | 0.637 | 0.625 | 34,563 | 1,049 | 35,612 | 2.52 |
|
||||
| **OVERALL** | **0.661** | **0.620** | **63,146** | **1,814** | **64,960** | **3.33** |
|
||||
|
||||
Memory Construction average token consumption (default agent, full build over 20 cases):
|
||||
|
||||
| Agent | input tok/case | output tok/case | total tok/case |
|
||||
|---|---|---|---|
|
||||
| default | 2,172,316 | 136,697 | 2,309,013 |
|
||||
|
||||
### 1M
|
||||
|
||||
agentscope==2.0.4.post1, conda reme env, 20 workers, full memory build
|
||||
(2026-08-05, 35 cases / 700 Qs, total 459.2 min)
|
||||
|
||||
| Type | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|
||||
|---|---|---|---|---|---|---|
|
||||
| abstention | 0.429 | 0.429 | 118,707 | 1,178 | 119,886 | 4.20 |
|
||||
| contradiction_resolution | 0.391 | 0.364 | 49,787 | 810 | 50,597 | 2.50 |
|
||||
| event_ordering | 0.558 | 0.456 | 201,514 | 3,889 | 205,403 | 4.79 |
|
||||
| information_extraction | 0.809 | 0.772 | 78,950 | 894 | 79,844 | 3.00 |
|
||||
| instruction_following | 0.852 | 0.832 | 55,757 | 924 | 56,681 | 2.81 |
|
||||
| knowledge_update | 0.779 | 0.771 | 45,981 | 665 | 46,646 | 2.37 |
|
||||
| multi_session_reasoning | 0.658 | 0.612 | 138,133 | 2,873 | 141,006 | 4.40 |
|
||||
| preference_following | 0.798 | 0.777 | 51,796 | 920 | 52,716 | 2.53 |
|
||||
| summarization | 0.693 | 0.537 | 158,794 | 2,905 | 161,700 | 4.44 |
|
||||
| temporal_reasoning | 0.536 | 0.536 | 100,176 | 3,148 | 103,324 | 3.90 |
|
||||
| **OVERALL** | **0.650** | **0.609** | **99,959** | **1,821** | **101,780** | **3.49** |
|
||||
|
||||
Memory Construction average token consumption (default agent, full build over 35 cases):
|
||||
|
||||
| Agent | input tok/case | output tok/case | total tok/case |
|
||||
|---|---|---|---|
|
||||
| default | 31,943,817 | 1,417,061 | 33,360,878 |
|
||||
119
benchmark/beam/README_ZH.md
Normal file
119
benchmark/beam/README_ZH.md
Normal file
|
|
@ -0,0 +1,119 @@
|
|||
# BEAM 评测
|
||||
|
||||
[English version](./README.md)
|
||||
|
||||
BEAM 是一个面向**长上下文对话场景**的记忆能力评测基准。每个 case 包含一段被切分为多个
|
||||
batch 的超长对话；ReMe 将每个 batch 转换为一个会话，按时间顺序摄入后，以 agentic（ReAct）
|
||||
模式回答探测问题。答案由 BEAM 基于 rubric 的 `answer_judge` 任务打分，同时给出分级分数与二元
|
||||
判定，并输出各类型平均分。
|
||||
|
||||
BEAM 按对话规模提供多种数据变体 —— `100K` / `500K` / `1M` / `10M`，可在不同上下文长度下
|
||||
压测记忆系统。题型包括 abstention（拒答）、contradiction resolution（矛盾消解）、event
|
||||
ordering（事件排序）、information extraction（信息抽取）、instruction following（指令遵循）、
|
||||
knowledge update（知识更新）、multi-session reasoning（多会话推理）、preference following
|
||||
（偏好遵循）、summarization（摘要）与 temporal reasoning（时间推理）。
|
||||
|
||||
> 公共设置（依赖、凭据、日志约定）见[总评测说明](../README_ZH.md)。
|
||||
|
||||
## 1. 获取数据集
|
||||
|
||||
BEAM 是公开仓库，clone 到 `benchmark/beam/dataset/` 下：
|
||||
|
||||
```bash
|
||||
mkdir -p benchmark/beam/dataset
|
||||
cd benchmark/beam/dataset
|
||||
git clone https://github.com/mohammadtavakoli78/BEAM.git
|
||||
```
|
||||
|
||||
clone 完成后，`benchmark/beam/dataset/BEAM/` 目录下应包含 `chats/`、`src/`、`topics/` 等子目录。
|
||||
|
||||
## 2. 运行
|
||||
|
||||
在仓库根目录执行：
|
||||
|
||||
```bash
|
||||
python benchmark/beam/run.py
|
||||
python benchmark/beam/run.py --config benchmark/beam/config.yaml
|
||||
python benchmark/beam/run.py -q # 安静模式
|
||||
python benchmark/beam/run.py --eval_only # 复用已有工作区，仅执行查询 + 评判
|
||||
```
|
||||
|
||||
## 3. 流程
|
||||
|
||||
1. 为每个 case 加载 `chat.json`，将每个 batch 转换为一个 ReMe 会话。
|
||||
2. 按时间顺序将会话摄入独立工作区，随后执行 `digest_update`。
|
||||
3. 以 agentic（ReAct）模式回答每个探测问题。
|
||||
4. 通过 BEAM 基于 rubric 的 `answer_judge` 任务打分，并输出各类型平均分。
|
||||
|
||||
## 4. 关键配置 —— `benchmark/beam/config.yaml`
|
||||
|
||||
| 配置项 | 含义 |
|
||||
| --- | --- |
|
||||
| `dataset.beam_root` | BEAM 数据集根目录（`benchmark/beam/dataset/BEAM`）。 |
|
||||
| `dataset.chat_size` | 运行的变体：`100K` / `500K` / `1M` / `10M`。 |
|
||||
| `dataset.case_ids` | 指定 case（如 `["1","2"]`），空表示全部。 |
|
||||
| `dataset.start_index` / `num_items` | case 分页（`num_items` 为 `0` 表示全部）。 |
|
||||
| `dataset.workspace_root` | case 工作区根目录（`benchmark/beam/workspaces/beam`）。 |
|
||||
| `evaluation.num_workers` | `0` = 自动，`1` = 串行，`>1` = 并行。 |
|
||||
| `reme.config` | 使用的 ReMe 配置（`beam.yaml`）。 |
|
||||
| `output.dir` | 结果目录（`benchmark/beam/results`）。 |
|
||||
|
||||
## 5. 输出
|
||||
|
||||
结果以 JSON 文件写入 `output.dir`，文件名为 `results_<chat_size>_<timestamp>.json`，
|
||||
同时控制台会打印含各类型分数的汇总。日志约定在各基准间通用，见
|
||||
[总说明](../README_ZH.md#输出与日志)。
|
||||
|
||||
## 6. 参考结果
|
||||
|
||||
> 以下结果使用 longmemeval 版本的 prompt。
|
||||
|
||||
### 100K
|
||||
|
||||
agentscope==2.0.4.post1，conda reme 环境，20 并发，eval-only（复用已构建 memory）
|
||||
（2026-08-05，20 cases / 400 Qs，总耗时 46.0 min）
|
||||
|
||||
| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|
||||
|---|---|---|---|---|---|---|
|
||||
| abstention | 0.550 | 0.550 | 96,031 | 1,070 | 97,101 | 4.58 |
|
||||
| contradiction_resolution | 0.438 | 0.412 | 32,263 | 872 | 33,135 | 2.48 |
|
||||
| event_ordering | 0.501 | 0.423 | 140,195 | 5,163 | 145,358 | 4.70 |
|
||||
| information_extraction | 0.873 | 0.832 | 50,245 | 883 | 51,128 | 3.15 |
|
||||
| instruction_following | 0.750 | 0.725 | 37,986 | 848 | 38,834 | 2.67 |
|
||||
| knowledge_update | 0.688 | 0.675 | 31,198 | 651 | 31,849 | 2.27 |
|
||||
| multi_session_reasoning | 0.626 | 0.584 | 85,038 | 4,563 | 89,601 | 4.28 |
|
||||
| preference_following | 0.925 | 0.912 | 34,281 | 989 | 35,270 | 2.50 |
|
||||
| summarization | 0.623 | 0.461 | 89,657 | 2,056 | 91,713 | 4.12 |
|
||||
| temporal_reasoning | 0.637 | 0.625 | 34,563 | 1,049 | 35,612 | 2.52 |
|
||||
| **OVERALL** | **0.661** | **0.620** | **63,146** | **1,814** | **64,960** | **3.33** |
|
||||
|
||||
Memory Construction 平均 token 消耗（default agent，20 cases 全量构建）：
|
||||
|
||||
| Agent | input tok/case | output tok/case | total tok/case |
|
||||
|---|---|---|---|
|
||||
| default | 2,172,316 | 136,697 | 2,309,013 |
|
||||
|
||||
### 1M
|
||||
|
||||
agentscope==2.0.4.post1，conda reme 环境，20 并发，全量构建 memory
|
||||
（2026-08-05，35 cases / 700 Qs，总耗时 459.2 min）
|
||||
|
||||
| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|
||||
|---|---|---|---|---|---|---|
|
||||
| abstention | 0.429 | 0.429 | 118,707 | 1,178 | 119,886 | 4.20 |
|
||||
| contradiction_resolution | 0.391 | 0.364 | 49,787 | 810 | 50,597 | 2.50 |
|
||||
| event_ordering | 0.558 | 0.456 | 201,514 | 3,889 | 205,403 | 4.79 |
|
||||
| information_extraction | 0.809 | 0.772 | 78,950 | 894 | 79,844 | 3.00 |
|
||||
| instruction_following | 0.852 | 0.832 | 55,757 | 924 | 56,681 | 2.81 |
|
||||
| knowledge_update | 0.779 | 0.771 | 45,981 | 665 | 46,646 | 2.37 |
|
||||
| multi_session_reasoning | 0.658 | 0.612 | 138,133 | 2,873 | 141,006 | 4.40 |
|
||||
| preference_following | 0.798 | 0.777 | 51,796 | 920 | 52,716 | 2.53 |
|
||||
| summarization | 0.693 | 0.537 | 158,794 | 2,905 | 161,700 | 4.44 |
|
||||
| temporal_reasoning | 0.536 | 0.536 | 100,176 | 3,148 | 103,324 | 3.90 |
|
||||
| **OVERALL** | **0.650** | **0.609** | **99,959** | **1,821** | **101,780** | **3.49** |
|
||||
|
||||
Memory Construction 平均 token 消耗（default agent，35 cases 全量构建）：
|
||||
|
||||
| Agent | input tok/case | output tok/case | total tok/case |
|
||||
|---|---|---|---|
|
||||
| default | 31,943,817 | 1,417,061 | 33,360,878 |
|
||||
96
benchmark/longmemeval/README.md
Normal file
96
benchmark/longmemeval/README.md
Normal file
|
|
@ -0,0 +1,96 @@
|
|||
[中文版 / Chinese version](./README_ZH.md)
|
||||
|
||||
# LongMemEval Benchmark
|
||||
|
||||
LongMemEval is a benchmark for **long-term memory over multi-session chat
|
||||
histories**. Each item provides a chronologically ordered set of chat sessions
|
||||
between a user and an assistant, followed by a probing question whose answer is
|
||||
only recoverable by reasoning over the user-owned memory. ReMe ingests the
|
||||
sessions into an isolated per-item workspace, answers the question via an
|
||||
agentic (ReAct) mode, and scores the answer with an LLM-as-judge.
|
||||
|
||||
Question types include single-session (user / assistant / preference),
|
||||
multi-session reasoning, knowledge update, and temporal reasoning.
|
||||
|
||||
> For the shared setup (dependencies, credentials, log conventions) see the
|
||||
> [top-level benchmark README](../README.md).
|
||||
|
||||
## 1. Get the Dataset
|
||||
|
||||
ReMe uses only the **cleaned-S** split, hosted on HuggingFace:
|
||||
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2).
|
||||
The download script fetches it via the hf-mirror.com mirror; to use a different
|
||||
mirror, modify `BASE_URL` in [`download.py`](./download.py).
|
||||
|
||||
```bash
|
||||
cd benchmark/longmemeval
|
||||
python download.py # saves dataset/longmemeval_s_reme_cleaned.json; skips if already present
|
||||
```
|
||||
|
||||
Ground truth is embedded in the data file.
|
||||
|
||||
## 2. Run
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
python benchmark/longmemeval/run.py
|
||||
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
|
||||
python benchmark/longmemeval/run.py -q # quiet: only eval-level logs
|
||||
python benchmark/longmemeval/run.py --log-level WARNING # reduce eval runner logs
|
||||
python benchmark/longmemeval/run.py --reme-log-level WARNING # reduce reme internal logs
|
||||
python benchmark/longmemeval/run.py --eval_only # reuse existing workspaces, query + judge only
|
||||
```
|
||||
|
||||
## 3. Pipeline
|
||||
|
||||
1. Load the dataset (ground truth is embedded in the data file).
|
||||
2. For each item, create an isolated workspace and ingest sessions in chronological order.
|
||||
3. Trigger `auto_dream` when consecutive sessions cross the configured hour (default 23:00).
|
||||
4. Answer each question via agentic (ReAct) mode.
|
||||
5. Judge the answer (binary yes/no) with the `answer_judge` job and print per-type accuracy.
|
||||
|
||||
## 4. Key config — `benchmark/longmemeval/config.yaml`
|
||||
|
||||
| Key | Meaning |
|
||||
| --- | --- |
|
||||
| `dataset.path` | Dataset file to evaluate (e.g. `longmemeval_s_reme_cleaned.json`); ground truth is included. |
|
||||
| `dataset.start_index` / `num_items` | Slice of items to evaluate. |
|
||||
| `dataset.question_types` | Filter by question type; empty = all. |
|
||||
| `dataset.workspace_root` | Per-item workspace root (`benchmark/longmemeval/workspaces/longmemeval-s`). |
|
||||
| `evaluation.num_workers` | `0` = auto (cpu-2), `1` = sequential, `>1` = parallel. |
|
||||
| `evaluation.filter_future_sessions` | Only ingest sessions with timestamp ≤ `question_date`. |
|
||||
| `reme.config` | ReMe config used (`lme.yaml`). |
|
||||
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | Dream triggering behavior. |
|
||||
| `output.dir` | Results directory (`benchmark/longmemeval/results`). |
|
||||
|
||||
## 5. Outputs
|
||||
|
||||
Results are JSON files written to `output.dir` as `results_<timestamp>.json`,
|
||||
with a per-type accuracy summary also printed to the console. Logging
|
||||
conventions are shared across benchmarks — see the
|
||||
[top-level README](../README.md#outputs--logs).
|
||||
|
||||
## 6. Reference Results
|
||||
|
||||
### cleaned-s
|
||||
|
||||
**Basic settings**
|
||||
|
||||
1. Modified auto-memory prompt, auto-dream disabled.
|
||||
2. All sessions in reme-memory are strictly earlier than the question time.
|
||||
|
||||
**Results**
|
||||
|
||||
agentscope==2.0.4.post1, conda reme env, 32 workers, eval-only (reusing prebuilt memory)
|
||||
(2026-08-06, 500 items, total 10.0 min)
|
||||
|
||||
| Type | Agentic | input tok/q | output tok/q | total tok/q | tool calls/q |
|
||||
|---|---|---|---|---|---|
|
||||
| knowledge-update | 0.910 | 31,581 | 589 | 32,169 | 2.90 |
|
||||
| multi-session | 0.842 | 52,837 | 1,474 | 54,311 | 4.21 |
|
||||
| single-session-assistant | 1.000 | 15,596 | 279 | 15,875 | 1.89 |
|
||||
| single-session-preference | 0.633 | 36,802 | 818 | 37,620 | 3.60 |
|
||||
| single-session-user | 0.986 | 27,433 | 359 | 27,792 | 2.60 |
|
||||
| temporal-reasoning | 0.902 | 62,674 | 985 | 63,659 | 4.97 |
|
||||
| **OVERALL** | **0.894** | **43,448** | **876** | **44,324** | **3.69** |
|
||||
90
benchmark/longmemeval/README_ZH.md
Normal file
90
benchmark/longmemeval/README_ZH.md
Normal file
|
|
@ -0,0 +1,90 @@
|
|||
# LongMemEval 评测
|
||||
|
||||
[English version](./README.md)
|
||||
|
||||
LongMemEval 是一个面向**多轮多会话历史的长期记忆能力**的评测基准。每个条目提供一组按时间
|
||||
顺序排列的用户与助手之间的会话，以及一个只能通过推理用户自有记忆才能回答的探测问题。ReMe
|
||||
将会话摄入按条目隔离的工作区，以 agentic（ReAct）模式回答问题，最后由 LLM-as-judge 打分。
|
||||
|
||||
题型包括单会话（user / assistant / preference）、多会话推理、知识更新与时间推理等。
|
||||
|
||||
> 公共设置（依赖、凭据、日志约定）见[总评测说明](../README_ZH.md)。
|
||||
|
||||
## 1. 获取数据集
|
||||
|
||||
ReMe 仅使用 **cleaned-S** 版本，数据托管在 HuggingFace：
|
||||
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)。
|
||||
下载脚本经 hf-mirror.com 镜像源获取，如需更换源请修改 [`download.py`](./download.py) 中的
|
||||
`BASE_URL`。
|
||||
|
||||
```bash
|
||||
cd benchmark/longmemeval
|
||||
python download.py # 保存为 dataset/longmemeval_s_reme_cleaned.json，已存在则自动跳过
|
||||
```
|
||||
|
||||
ground truth 已内嵌在数据文件中。
|
||||
|
||||
## 2. 运行
|
||||
|
||||
在仓库根目录执行：
|
||||
|
||||
```bash
|
||||
python benchmark/longmemeval/run.py
|
||||
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
|
||||
python benchmark/longmemeval/run.py -q # 安静模式：仅评测级日志
|
||||
python benchmark/longmemeval/run.py --log-level WARNING # 降低评测 runner 日志
|
||||
python benchmark/longmemeval/run.py --reme-log-level WARNING # 降低 reme 内部日志
|
||||
python benchmark/longmemeval/run.py --eval_only # 复用已有工作区，仅执行查询 + 评判
|
||||
```
|
||||
|
||||
## 3. 流程
|
||||
|
||||
1. 加载数据集（ground truth 已内嵌在数据文件中）。
|
||||
2. 为每个条目创建独立工作区，按时间顺序摄入会话。
|
||||
3. 当相邻会话跨越配置的时刻（默认 23:00）时触发 `auto_dream`。
|
||||
4. 以 agentic（ReAct）模式回答每个问题。
|
||||
5. 通过 `answer_judge` 任务对答案做二元（yes/no）评判，并输出各类型准确率。
|
||||
|
||||
## 4. 关键配置 —— `benchmark/longmemeval/config.yaml`
|
||||
|
||||
| 配置项 | 含义 |
|
||||
| --- | --- |
|
||||
| `dataset.path` | 待评测的数据集文件（如 `longmemeval_s_reme_cleaned.json`），已包含 ground truth。 |
|
||||
| `dataset.start_index` / `num_items` | 评测条目的切片范围。 |
|
||||
| `dataset.question_types` | 按问题类型过滤，空表示全部。 |
|
||||
| `dataset.workspace_root` | 条目工作区根目录（`benchmark/longmemeval/workspaces/longmemeval-s`）。 |
|
||||
| `evaluation.num_workers` | `0` = 自动（cpu-2），`1` = 串行，`>1` = 并行。 |
|
||||
| `evaluation.filter_future_sessions` | 仅摄入时间戳 ≤ `question_date` 的会话。 |
|
||||
| `reme.config` | 使用的 ReMe 配置（`lme.yaml`）。 |
|
||||
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | dream 触发行为。 |
|
||||
| `output.dir` | 结果目录（`benchmark/longmemeval/results`）。 |
|
||||
|
||||
## 5. 输出
|
||||
|
||||
结果以 JSON 文件写入 `output.dir`，文件名为 `results_<timestamp>.json`，
|
||||
同时控制台会打印含各类型准确率的汇总。日志约定在各基准间通用，见
|
||||
[总说明](../README_ZH.md#输出与日志)。
|
||||
|
||||
## 6. 参考结果
|
||||
|
||||
### cleaned-s
|
||||
|
||||
**基础设置**
|
||||
|
||||
1. 使用修改后的 auto-memory prompt，关闭 auto-dream 机制
|
||||
2. reme-memory 中的全部 session 的时间一定早于 question 的时间
|
||||
|
||||
**结果**
|
||||
|
||||
agentscope==2.0.4.post1, conda reme env, 32 workers, eval-only（复用预构建记忆）
|
||||
（2026-08-06，500 题，总计 10.0 min）
|
||||
|
||||
| 类型 | Agentic | input tok/q | output tok/q | total tok/q | tool calls/q |
|
||||
|---|---|---|---|---|---|
|
||||
| knowledge-update | 0.910 | 31,581 | 589 | 32,169 | 2.90 |
|
||||
| multi-session | 0.842 | 52,837 | 1,474 | 54,311 | 4.21 |
|
||||
| single-session-assistant | 1.000 | 15,596 | 279 | 15,875 | 1.89 |
|
||||
| single-session-preference | 0.633 | 36,802 | 818 | 37,620 | 3.60 |
|
||||
| single-session-user | 0.986 | 27,433 | 359 | 27,792 | 2.60 |
|
||||
| temporal-reasoning | 0.902 | 62,674 | 985 | 63,659 | 4.97 |
|
||||
| **OVERALL** | **0.894** | **43,448** | **876** | **44,324** | **3.69** |
|
||||
76
benchmark/longmemeval/kill.sh
Normal file
76
benchmark/longmemeval/kill.sh
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
#!/bin/bash
|
||||
# 杀死指定进程及其所有子进程
|
||||
# Usage: bash kill.sh <PID>
|
||||
|
||||
if [ -z "$1" ]; then
|
||||
echo "Usage: bash kill.sh <PID>"
|
||||
echo " 杀死指定进程及其所有子进程"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
PID=$1
|
||||
|
||||
# 检查进程是否存在
|
||||
if ! kill -0 "$PID" 2>/dev/null; then
|
||||
echo "进程 $PID 不存在"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 递归收集所有子进程（包括子进程的子进程）
|
||||
collect_children() {
|
||||
local parent=$1
|
||||
local children
|
||||
children=$(ps -o pid= --ppid "$parent" 2>/dev/null | tr -d ' ')
|
||||
for child in $children; do
|
||||
collect_children "$child"
|
||||
done
|
||||
echo "$parent"
|
||||
}
|
||||
|
||||
# 收集进程树（子进程在前，父进程在后，保证先杀子再杀父）
|
||||
PROCESS_TREE=$(collect_children "$PID")
|
||||
TOTAL=$(echo "$PROCESS_TREE" | wc -l | tr -d ' ')
|
||||
|
||||
echo "进程树（共 $TOTAL 个进程）:"
|
||||
while read -r p; do
|
||||
cmd=$(ps -o args= -p "$p" 2>/dev/null | head -c 80)
|
||||
printf " PID=%-8s %s\n" "$p" "$cmd"
|
||||
done <<< "$PROCESS_TREE"
|
||||
|
||||
# 先 SIGTERM 优雅终止
|
||||
echo ""
|
||||
echo "发送 SIGTERM..."
|
||||
while read -r p; do
|
||||
kill "$p" 2>/dev/null
|
||||
done <<< "$PROCESS_TREE"
|
||||
|
||||
# 等待最多 5 秒
|
||||
for i in $(seq 1 5); do
|
||||
alive=false
|
||||
while read -r p; do
|
||||
if kill -0 "$p" 2>/dev/null; then
|
||||
alive=true
|
||||
fi
|
||||
done <<< "$PROCESS_TREE"
|
||||
if [ "$alive" = false ]; then
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
# 检查是否还有残留，强制 SIGKILL
|
||||
remaining=false
|
||||
while read -r p; do
|
||||
if kill -0 "$p" 2>/dev/null; then
|
||||
remaining=true
|
||||
fi
|
||||
done <<< "$PROCESS_TREE"
|
||||
|
||||
if [ "$remaining" = true ]; then
|
||||
echo "部分进程未响应，发送 SIGKILL..."
|
||||
while read -r p; do
|
||||
kill -9 "$p" 2>/dev/null
|
||||
done <<< "$PROCESS_TREE"
|
||||
fi
|
||||
|
||||
echo "已终止进程树（根 PID=$PID，共 $TOTAL 个进程）"
|
||||
|
|
@ -1,54 +0,0 @@
|
|||
# beam result
|
||||
|
||||
## longmemeval版本的prompt
|
||||
|
||||
### 100K
|
||||
|
||||
|
||||
agentscope==2.0.4.post1, conda reme 环境, 20 并发, eval-only（复用已构建 memory）
|
||||
(2026-08-05, 20 cases / 400 Qs, 总耗时 46.0 min)
|
||||
|
||||
| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|
||||
|---|---|---|---|---|---|---|
|
||||
| abstention | 0.550 | 0.550 | 96,031 | 1,070 | 97,101 | 4.58 |
|
||||
| contradiction_resolution | 0.438 | 0.412 | 32,263 | 872 | 33,135 | 2.48 |
|
||||
| event_ordering | 0.501 | 0.423 | 140,195 | 5,163 | 145,358 | 4.70 |
|
||||
| information_extraction | 0.873 | 0.832 | 50,245 | 883 | 51,128 | 3.15 |
|
||||
| instruction_following | 0.750 | 0.725 | 37,986 | 848 | 38,834 | 2.67 |
|
||||
| knowledge_update | 0.688 | 0.675 | 31,198 | 651 | 31,849 | 2.27 |
|
||||
| multi_session_reasoning | 0.626 | 0.584 | 85,038 | 4,563 | 89,601 | 4.28 |
|
||||
| preference_following | 0.925 | 0.912 | 34,281 | 989 | 35,270 | 2.50 |
|
||||
| summarization | 0.623 | 0.461 | 89,657 | 2,056 | 91,713 | 4.12 |
|
||||
| temporal_reasoning | 0.637 | 0.625 | 34,563 | 1,049 | 35,612 | 2.52 |
|
||||
| **OVERALL** | **0.661** | **0.620** | **63,146** | **1,814** | **64,960** | **3.33** |
|
||||
|
||||
Memory Construction 平均 token 消耗（default agent, 20 cases 全量构建）:
|
||||
|
||||
| Agent | input tok/case | output tok/case | total tok/case |
|
||||
|---|---|---|---|
|
||||
| default | 2,172,316 | 136,697 | 2,309,013 |
|
||||
|
||||
### 1M
|
||||
|
||||
agentscope==2.0.4.post1, conda reme 环境, 20 并发, 全量构建 memory
|
||||
(2026-08-05, 35 cases / 700 Qs, 总耗时 459.2 min)
|
||||
|
||||
| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|
||||
|---|---|---|---|---|---|---|
|
||||
| abstention | 0.429 | 0.429 | 118,707 | 1,178 | 119,886 | 4.20 |
|
||||
| contradiction_resolution | 0.391 | 0.364 | 49,787 | 810 | 50,597 | 2.50 |
|
||||
| event_ordering | 0.558 | 0.456 | 201,514 | 3,889 | 205,403 | 4.79 |
|
||||
| information_extraction | 0.809 | 0.772 | 78,950 | 894 | 79,844 | 3.00 |
|
||||
| instruction_following | 0.852 | 0.832 | 55,757 | 924 | 56,681 | 2.81 |
|
||||
| knowledge_update | 0.779 | 0.771 | 45,981 | 665 | 46,646 | 2.37 |
|
||||
| multi_session_reasoning | 0.658 | 0.612 | 138,133 | 2,873 | 141,006 | 4.40 |
|
||||
| preference_following | 0.798 | 0.777 | 51,796 | 920 | 52,716 | 2.53 |
|
||||
| summarization | 0.693 | 0.537 | 158,794 | 2,905 | 161,700 | 4.44 |
|
||||
| temporal_reasoning | 0.536 | 0.536 | 100,176 | 3,148 | 103,324 | 3.90 |
|
||||
| **OVERALL** | **0.650** | **0.609** | **99,959** | **1,821** | **101,780** | **3.49** |
|
||||
|
||||
Memory Construction 平均 token 消耗（default agent, 35 cases 全量构建）:
|
||||
|
||||
| Agent | input tok/case | output tok/case | total tok/case |
|
||||
|---|---|---|---|
|
||||
| default | 31,943,817 | 1,417,061 | 33,360,878 |
|
||||
|
|
@ -1,92 +0,0 @@
|
|||
# LongMemEval 数据集测试结果
|
||||
|
||||
## cleaned-s
|
||||
|
||||
**basic settings**
|
||||
|
||||
1. 使用修改后的auto-memory prompt，关闭auto-dream机制
|
||||
2. reme-memory中的全部session的时间一定早于question的时间
|
||||
|
||||
**results **
|
||||
|
||||
1. Agentic answer框架回答，每次最多调用5次search
|
||||
|
||||
| Category | Total | Correct | Wrong | Accuracy |
|
||||
|---|---|---|---|---|
|
||||
| single-session-user | 70 | 66 | 4 | 94.3% |
|
||||
| single-session-assistant | 56 | 52 | 4 | 92.9% |
|
||||
| knowledge-update | 78 | 60 | 18 | 76.9% |
|
||||
| multi-session | 133 | 93 | 40 | 69.9% |
|
||||
| temporal-reasoning | 133 | 78 | 55 | 58.6% |
|
||||
| single-session-preference | 30 | 8 | 22 | 26.7% |
|
||||
| **Overall** | **500** | **357** | **143** | **71.4%** |
|
||||
|
||||
2. prompted-based amswer，每次固定使用原始query召回10个fileChunk
|
||||
|
||||
| Category | Total | Correct | Wrong | Accuracy |
|
||||
|---|---|---|---|---|
|
||||
| single-session-assistant | 56 | 56 | 0 | 100.0% |
|
||||
| single-session-user | 70 | 67 | 3 | 95.7% |
|
||||
| knowledge-update | 78 | 69 | 9 | 88.5% |
|
||||
| multi-session | 133 | 99 | 34 | 74.4% |
|
||||
| temporal-reasoning | 133 | 83 | 50 | 62.4% |
|
||||
| single-session-preference | 30 | 16 | 14 | 53.3% |
|
||||
| **Overall** | **500** | **390** | **110** | **78.0%** |
|
||||
|
||||
3. golden session。 使用与prompt-based answer相似的方法，唯一区别是，输入的chunk是longMemEval提供的golden session。
|
||||
|
||||
| Category | Total | Correct | Wrong | Accuracy |
|
||||
|---|---|---|---|---|
|
||||
| single-session-assistant | 56 | 56 | 0 | 100.0% |
|
||||
| single-session-user | 70 | 69 | 1 | 98.6% |
|
||||
| knowledge-update | 78 | 74 | 4 | 94.9% |
|
||||
| temporal-reasoning | 133 | 124 | 9 | 93.2% |
|
||||
| multi-session | 133 | 117 | 16 | 88.0% |
|
||||
| single-session-preference | 30 | 17 | 13 | 56.7% |
|
||||
| **Overall** | **500** | **457** | **43** | **91.4%** |
|
||||
|
||||
4. golden session + time filter. 和上面一个实验的区别是，输入的golden被过滤了一次，要求输入session的时间戳必须早于question的时间才行。
|
||||
|
||||
一共被过滤掉了75个session，44个question受到了影响。temperal reasoning类型受影响最大。有20个case不包含任何一个groundtruth session。 根据golden session回答正确并且golden session非空，一共有424个case。
|
||||
|
||||
| Category | Total | Correct | Wrong | Accuracy |
|
||||
|---|---|---|---|---|
|
||||
| knowledge-update | 78 | 75 | 3 | 96.2% |
|
||||
| single-session-user | 70 | 67 | 3 | 95.7% |
|
||||
| multi-session | 133 | 122 | 11 | 91.7% |
|
||||
| single-session-assistant | 56 | 55 | 1 | 98.2% |
|
||||
| temporal-reasoning | 133 | 91 | 42 | 68.4% |
|
||||
| single-session-preference | 30 | 16 | 14 | 53.3% |
|
||||
| **Overall** | **500** | **426** | **74** | **85.2%** |
|
||||
|
||||
5. 关闭auto-memory机制，根据原始query一次性混合检索召回原始session，计算recall.
|
||||
|
||||
| Category | Total | yes-judge | recall@5 / yes | recall@10 / yes |
|
||||
|---|---|---|---|---|
|
||||
| knowledge-update | 78 | 75 | 99.3% | 100% |
|
||||
| single-session-user | 70 | 67 | 100% | 100% |
|
||||
| multi-session | 133 | 122 | 91.8% | 95.8% |
|
||||
| single-session-assistant | 56 | 55 | 100% | 100% |
|
||||
| temporal-reasoning | 133 | 91 | 87.6% | 94.2% |
|
||||
| single-session-preference | 30 | 16 | 100% | 100% |
|
||||
| **Overall** | **500** | **426** | **87.6%** | **94.2%** |
|
||||
|
||||
|
||||
## 最终groundtruth
|
||||
|
||||
### agentic + prompted（最终GT，2026-07-16）
|
||||
|
||||
|
||||
| Category | Total | Agentic | Prompted limit=15 |
|
||||
|---|---|---|---|
|
||||
| single-session-assistant | 56 | 56/56 (100.0%) | 54/56 (96.4%) |
|
||||
| single-session-user | 70 | 66/70 (94.3%) | 62/70 (88.6%) |
|
||||
| knowledge-update | 78 | 75/78 (96.2%) | 67/78 (85.9%) |
|
||||
| temporal-reasoning | 133 | 122/133 (91.7%) | 117/133 (88.0%) |
|
||||
| multi-session | 133 | 115/133 (86.5%) | 101/133 (75.9%) |
|
||||
| single-session-preference | 30 | 21/30 (70.0%) | 10/30 (33.3%) |
|
||||
| **Overall** | **500** | **455/500 (91.0%)** | **411/500 (82.2%)** |
|
||||
|
||||
Prompted token 消耗：总 input 13,111,421 (平均 26,275/题)，总 output 313,370 (平均 628/题)。
|
||||
平均 sessions_ingested: 44.8，dreams_triggered: 0。
|
||||
|
||||
|
|
@ -1,25 +1,16 @@
|
|||
system_prompt: |
|
||||
You are a memory retrieval assistant. A memory system manages the user's conversation history. Given a query, you MUST call the `search` tool at least once to retrieve relevant content from the memory system before answering.
|
||||
|
||||
## Memory System
|
||||
The memory system stores the user's history in three forms:
|
||||
- **Summary notes**: condensed knowledge distilled from past sessions.
|
||||
- **Compressed session chunks**: fragments of session transcripts compressed to drop low-information phrasing; the dialogue structure is generally kept, but some detail may be shortened or lost.
|
||||
- **Original sessions**: complete verbatim transcripts of past conversations, stored as memory files.
|
||||
|
||||
The `search` tool retrieves summary notes and compressed session chunks; it does NOT return original sessions directly. Every result is a chunk — a fragment of a memory file identified by its path — and may lack surrounding context. Compressed session chunks are marked with a leading `compressed session chunk:` notice. When a chunk looks highly relevant to the query but its content is vague, truncated, or missing detail due to compression, use the `read` tool with the chunk's path to read the original session before drawing conclusions.
|
||||
|
||||
## Workflow
|
||||
1. Use `search` to locate possibly relevant content. It is a hybrid retrieval tool combining BM25 keyword matching and vector semantic search, ensuring diverse recall. Call it multiple times with different query phrasings to gather comprehensive information; do not stop after the first relevant result unless you are confident the information is sufficient.
|
||||
2. When a retrieved chunk is strongly related to the query but ambiguous because of compression, call `read` on its path to recover the full original session, and ground your answer on that.
|
||||
3. Use `add_draft` to save key findings during the search, and `read_all_draft` to review all saved notes before answering.
|
||||
4. After finding a preliminary answer, continue searching from multiple angles to check for potentially conflicting, supplementary, or updated information. The goal is to improve the accuracy and completeness of the final answer.
|
||||
5. Stop searching and generate the final answer once you are confident that further searches will not yield additional useful information.
|
||||
- Your total tool calls should be at most 9 times.
|
||||
You are a memory retrieval assistant. You MUST use the search tool to find information before answering.
|
||||
- Your total time of tool calls should be at most 9 times
|
||||
|
||||
## Search Strategy
|
||||
- You can call 'search' tool to search multiple times (at least once) with different queries to gather comprehensive information.
|
||||
- Do not stop after the first relevant one unless you are confident that the information is sufficient.
|
||||
- You can use 'read' tool to read specific parts of a specific file.
|
||||
- After finding a preliminary answer, continue searching from multiple angles to check for potentially conflicting, supplementary, or updated information.
|
||||
## Draft Tool
|
||||
- Use 'add_draft' to save key findings during search, and 'read_all_draft' to review all saved notes before answering.
|
||||
## Answer Rules
|
||||
- Answer based ONLY on retrieved or read context.
|
||||
- If memories contain contradictory information, weigh the context, query, and most importantly the timestamp of each memory to determine the final answer.
|
||||
- Answer based ONLY on retrieved context.
|
||||
- Output ONLY the direct factual answer — no reasoning, no search process, no elaboration.
|
||||
- Do NOT invent or assume information that isn't in the memories. Do NOT extrapolate or fabricate connections from related but non-direct evidence.
|
||||
- If information is not found or not sufficient after multiple searches, or if the maximum iteration count is reached without finding answer, reply: 'Information not found.'
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue