mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-11-12 08:45:15 +00:00
Merge upstream/main into sandbox
This commit is contained in:
酱牛肉
commit
e4215c93b2
31 changed files with 895 additions and 584 deletions
|
|
@ -1,155 +0,0 @@
|
|||
[中文版 / Chinese version](./README_ZH.md)
|
||||
|
||||
# ReMe Benchmarks
|
||||
|
||||
Reproduction guide for the two memory benchmarks shipped with ReMe:
|
||||
|
||||
- **LongMemEval** — long-term memory over multi-session chat histories.
|
||||
- **BEAM** — memory capability over long-context chat cases with rubric-based judging.
|
||||
|
||||
Each benchmark runs its own end-to-end pipeline: ingest sessions into an isolated
|
||||
per-item workspace, answer probing questions via an agentic (ReAct) mode,
|
||||
then score answers with an LLM-as-judge.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
Install ReMe with dev + core extras (Python 3.11+):
|
||||
|
||||
```bash
|
||||
pip install -e ".[dev,core]"
|
||||
```
|
||||
|
||||
Configure model credentials in a project-root `.env` file (copied from `example.env`).
|
||||
The runners auto-load `.env` from the repository root. Required variables typically include:
|
||||
|
||||
```bash
|
||||
LLM_API_KEY=...
|
||||
LLM_BASE_URL=...
|
||||
EMBEDDING_API_KEY=...
|
||||
EMBEDDING_BASE_URL=...
|
||||
```
|
||||
|
||||
Model names and component wiring live in the ReMe configs referenced by each benchmark
|
||||
(`reme/config/lme.yaml` and `reme/config/beam.yaml`).
|
||||
|
||||
## 2. Download Datasets
|
||||
|
||||
Each benchmark keeps its own data under its directory:
|
||||
`benchmark/<name>/dataset` (input data), `benchmark/<name>/workspaces`
|
||||
(per-item memory workspaces), and `benchmark/<name>/results` (evaluation outputs).
|
||||
All three are excluded from Git.
|
||||
|
||||
**LongMemEval** — ReMe uses only the **cleaned-S** split, hosted on HuggingFace:
|
||||
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)
|
||||
(the script downloads via the hf-mirror.com mirror; to use a different mirror,
|
||||
modify `BASE_URL` in `download.py`):
|
||||
|
||||
```bash
|
||||
cd benchmark/longmemeval
|
||||
python download.py # saves dataset/longmemeval_s_reme_cleaned.json; skips if already present
|
||||
```
|
||||
|
||||
**BEAM** (public repository, cloned into `benchmark/beam/dataset/`):
|
||||
|
||||
```bash
|
||||
mkdir -p benchmark/beam/dataset
|
||||
cd benchmark/beam/dataset
|
||||
git clone https://github.com/mohammadtavakoli78/BEAM.git
|
||||
```
|
||||
|
||||
After cloning, `benchmark/beam/dataset/BEAM/` should contain `chats/`, `src/`,
|
||||
`topics/` and other subdirectories.
|
||||
|
||||
## 3. Run LongMemEval
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
python benchmark/longmemeval/run.py
|
||||
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
|
||||
python benchmark/longmemeval/run.py -q # quiet: only eval-level logs
|
||||
python benchmark/longmemeval/run.py --log-level WARNING # reduce eval runner logs
|
||||
python benchmark/longmemeval/run.py --reme-log-level WARNING # reduce reme internal logs
|
||||
python benchmark/longmemeval/run.py --eval_only # reuse existing workspaces, query + judge only
|
||||
```
|
||||
|
||||
### Pipeline
|
||||
|
||||
1. Load the dataset (ground truth is embedded in the data file).
|
||||
2. For each item, create an isolated workspace and ingest sessions in chronological order.
|
||||
3. Trigger `auto_dream` when consecutive sessions cross the configured hour (default 23:00).
|
||||
4. Answer each question via agentic (ReAct) mode.
|
||||
5. Judge the answer (binary yes/no) with the `answer_judge` job and print per-type accuracy.
|
||||
|
||||
### Key config — `benchmark/longmemeval/config.yaml`
|
||||
|
||||
| Key | Meaning |
|
||||
| --- | --- |
|
||||
| `dataset.path` | Dataset file to evaluate (e.g. `longmemeval_s_reme_cleaned.json`); ground truth is included. |
|
||||
| `dataset.start_index` / `num_items` | Slice of items to evaluate. |
|
||||
| `dataset.question_types` | Filter by question type; empty = all. |
|
||||
| `dataset.workspace_root` | Per-item workspace root (`benchmark/longmemeval/workspaces/longmemeval-s`). |
|
||||
| `evaluation.num_workers` | `0` = auto (cpu-2), `1` = sequential, `>1` = parallel. |
|
||||
| `evaluation.filter_future_sessions` | Only ingest sessions with timestamp ≤ `question_date`. |
|
||||
| `reme.config` | ReMe config used (`lme.yaml`). |
|
||||
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | Dream triggering behavior. |
|
||||
| `output.dir` | Results directory (`benchmark/longmemeval/results`). |
|
||||
|
||||
## 4. Run BEAM
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
python benchmark/beam/run.py
|
||||
python benchmark/beam/run.py --config benchmark/beam/config.yaml
|
||||
python benchmark/beam/run.py -q # quiet
|
||||
python benchmark/beam/run.py --eval_only # reuse existing workspaces, query + judge only
|
||||
```
|
||||
|
||||
### Pipeline
|
||||
|
||||
1. For each case, load `chat.json` and convert each batch into a ReMe session.
|
||||
2. Ingest sessions in chronological order into an isolated workspace, then `digest_update`.
|
||||
3. Answer each probing question via agentic (ReAct) mode.
|
||||
4. Score answers with BEAM's rubric-based `answer_judge` job and print per-type averages.
|
||||
|
||||
### Key config — `benchmark/beam/config.yaml`
|
||||
|
||||
| Key | Meaning |
|
||||
| --- | --- |
|
||||
| `dataset.beam_root` | BEAM dataset root (`benchmark/beam/dataset/BEAM`). |
|
||||
| `dataset.chat_size` | Variant to run: `100K` / `500K` / `1M` / `10M`. |
|
||||
| `dataset.case_ids` | Specific cases (e.g. `["1","2"]`); empty = all cases. |
|
||||
| `dataset.start_index` / `num_items` | Case pagination (`num_items` `0` = all). |
|
||||
| `dataset.workspace_root` | Per-case workspace root (`benchmark/beam/workspaces/beam`). |
|
||||
| `evaluation.num_workers` | `0` = auto, `1` = sequential, `>1` = parallel. |
|
||||
| `reme.config` | ReMe config used (`beam.yaml`). |
|
||||
| `output.dir` | Results directory (`benchmark/beam/results`). |
|
||||
|
||||
## 5. Outputs & Logs
|
||||
|
||||
- **Results**: JSON files written to `output.dir`
|
||||
(`results_<timestamp>.json` for LongMemEval,
|
||||
`results_<chat_size>_<timestamp>.json` for BEAM). A summary with per-type
|
||||
accuracy/score is also printed to the console.
|
||||
- **Logs**: when `output.log_to_file` is enabled, per-run logs are written to
|
||||
`logs/<log_prefix>_<timestamp>/` (a `runner.log` plus one `worker-<pid>.log`
|
||||
per worker process).
|
||||
|
||||
## 6. Stopping a Run
|
||||
|
||||
Parallel runs spawn a process tree. To terminate a run and all its workers cleanly:
|
||||
|
||||
```bash
|
||||
bash benchmark/kill.sh <PID>
|
||||
```
|
||||
|
||||
The script gracefully sends `SIGTERM` to the whole process tree, then escalates to
|
||||
`SIGKILL` for any process that does not exit within 5 seconds.
|
||||
|
||||
## 7. Reference Results
|
||||
|
||||
Recorded evaluation results are available in:
|
||||
|
||||
- [`longmemeval.md`](./results_md/longmemeval.md)
|
||||
- [`beam.md`](./results_md/beam.md)
|
||||
|
|
@ -1,149 +0,0 @@
|
|||
# ReMe 评测复现说明
|
||||
|
||||
ReMe 内置两个记忆能力评测基准的复现指南：
|
||||
|
||||
- **LongMemEval** —— 面向多轮多会话历史的长期记忆能力评测。
|
||||
- **BEAM** —— 面向长上下文对话场景、基于评分细则（rubric）打分的记忆能力评测。
|
||||
|
||||
每个基准都会运行完整的端到端流程：将会话摄入独立的按条目隔离的工作区，
|
||||
以 agentic（ReAct）模式回答探测问题，
|
||||
最后由 LLM-as-judge 对答案进行打分。
|
||||
|
||||
## 1. 环境准备
|
||||
|
||||
安装 ReMe 及 dev + core 附加依赖（Python 3.11+）：
|
||||
|
||||
```bash
|
||||
pip install -e ".[dev,core]"
|
||||
```
|
||||
|
||||
在项目根目录配置 `.env`（可从 `example.env` 复制），运行脚本会自动从仓库根目录加载 `.env`。
|
||||
通常需要以下变量：
|
||||
|
||||
```bash
|
||||
LLM_API_KEY=...
|
||||
LLM_BASE_URL=...
|
||||
EMBEDDING_API_KEY=...
|
||||
EMBEDDING_BASE_URL=...
|
||||
```
|
||||
|
||||
模型名称与组件装配位于各基准引用的 ReMe 配置中
|
||||
（`reme/config/lme.yaml` 与 `reme/config/beam.yaml`）。
|
||||
|
||||
## 2. 下载数据集
|
||||
|
||||
每个基准的数据都存放在各自目录下：`benchmark/<name>/dataset`（输入数据）、
|
||||
`benchmark/<name>/workspaces`（按条目隔离的记忆工作区）、
|
||||
`benchmark/<name>/results`（评测输出）。三者均不纳入 Git 版本管理。
|
||||
|
||||
**LongMemEval** —— ReMe 仅使用 **cleaned-S** 版本，数据托管在 HuggingFace：
|
||||
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)
|
||||
（下载脚本经 hf-mirror.com 镜像源获取，如需更换源请修改 `download.py` 中的 `BASE_URL`）：
|
||||
|
||||
```bash
|
||||
cd benchmark/longmemeval
|
||||
python download.py # 保存为 dataset/longmemeval_s_reme_cleaned.json，已存在则自动跳过
|
||||
```
|
||||
|
||||
**BEAM**（公开仓库，clone 到 `benchmark/beam/dataset/` 下）：
|
||||
|
||||
```bash
|
||||
mkdir -p benchmark/beam/dataset
|
||||
cd benchmark/beam/dataset
|
||||
git clone https://github.com/mohammadtavakoli78/BEAM.git
|
||||
```
|
||||
|
||||
clone 完成后，`benchmark/beam/dataset/BEAM/` 目录下应包含 `chats/`、`src/`、`topics/` 等子目录。
|
||||
|
||||
## 3. 运行 LongMemEval
|
||||
|
||||
在仓库根目录执行：
|
||||
|
||||
```bash
|
||||
python benchmark/longmemeval/run.py
|
||||
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
|
||||
python benchmark/longmemeval/run.py -q # 安静模式：仅评测级日志
|
||||
python benchmark/longmemeval/run.py --log-level WARNING # 降低评测 runner 日志
|
||||
python benchmark/longmemeval/run.py --reme-log-level WARNING # 降低 reme 内部日志
|
||||
python benchmark/longmemeval/run.py --eval_only # 复用已有工作区，仅执行查询 + 评判
|
||||
```
|
||||
|
||||
### 流程
|
||||
|
||||
1. 加载数据集（ground truth 已内嵌在数据文件中）。
|
||||
2. 为每个条目创建独立工作区，按时间顺序摄入会话。
|
||||
3. 当相邻会话跨越配置的时刻（默认 23:00）时触发 `auto_dream`。
|
||||
4. 以 agentic（ReAct）模式回答每个问题。
|
||||
5. 通过 `answer_judge` 任务对答案做二元（yes/no）评判，并输出各类型准确率。
|
||||
|
||||
### 关键配置 —— `benchmark/longmemeval/config.yaml`
|
||||
|
||||
| 配置项 | 含义 |
|
||||
| --- | --- |
|
||||
| `dataset.path` | 待评测的数据集文件（如 `longmemeval_s_reme_cleaned.json`），已包含 ground truth。 |
|
||||
| `dataset.start_index` / `num_items` | 评测条目的切片范围。 |
|
||||
| `dataset.question_types` | 按问题类型过滤，空表示全部。 |
|
||||
| `dataset.workspace_root` | 条目工作区根目录（`benchmark/longmemeval/workspaces/longmemeval-s`）。 |
|
||||
| `evaluation.num_workers` | `0` = 自动（cpu-2），`1` = 串行，`>1` = 并行。 |
|
||||
| `evaluation.filter_future_sessions` | 仅摄入时间戳 ≤ `question_date` 的会话。 |
|
||||
| `reme.config` | 使用的 ReMe 配置（`lme.yaml`）。 |
|
||||
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | dream 触发行为。 |
|
||||
| `output.dir` | 结果目录（`benchmark/longmemeval/results`）。 |
|
||||
|
||||
## 4. 运行 BEAM
|
||||
|
||||
在仓库根目录执行：
|
||||
|
||||
```bash
|
||||
python benchmark/beam/run.py
|
||||
python benchmark/beam/run.py --config benchmark/beam/config.yaml
|
||||
python benchmark/beam/run.py -q # 安静模式
|
||||
python benchmark/beam/run.py --eval_only # 复用已有工作区，仅执行查询 + 评判
|
||||
```
|
||||
|
||||
### 流程
|
||||
|
||||
1. 为每个 case 加载 `chat.json`，将每个 batch 转换为一个 ReMe 会话。
|
||||
2. 按时间顺序将会话摄入独立工作区，随后执行 `digest_update`。
|
||||
3. 以 agentic（ReAct）模式回答每个探测问题。
|
||||
4. 通过 BEAM 基于 rubric 的 `answer_judge` 任务打分，并输出各类型平均分。
|
||||
|
||||
### 关键配置 —— `benchmark/beam/config.yaml`
|
||||
|
||||
| 配置项 | 含义 |
|
||||
| --- | --- |
|
||||
| `dataset.beam_root` | BEAM 数据集根目录（`benchmark/beam/dataset/BEAM`）。 |
|
||||
| `dataset.chat_size` | 运行的变体：`100K` / `500K` / `1M` / `10M`。 |
|
||||
| `dataset.case_ids` | 指定 case（如 `["1","2"]`），空表示全部。 |
|
||||
| `dataset.start_index` / `num_items` | case 分页（`num_items` 为 `0` 表示全部）。 |
|
||||
| `dataset.workspace_root` | case 工作区根目录（`benchmark/beam/workspaces/beam`）。 |
|
||||
| `evaluation.num_workers` | `0` = 自动，`1` = 串行，`>1` = 并行。 |
|
||||
| `reme.config` | 使用的 ReMe 配置（`beam.yaml`）。 |
|
||||
| `output.dir` | 结果目录（`benchmark/beam/results`）。 |
|
||||
|
||||
## 5. 输出与日志
|
||||
|
||||
- **结果**：JSON 文件写入 `output.dir`
|
||||
（LongMemEval 为 `results_<timestamp>.json`，
|
||||
BEAM 为 `results_<chat_size>_<timestamp>.json`）。同时控制台会打印含各类型
|
||||
准确率/分数的汇总。
|
||||
- **日志**：当 `output.log_to_file` 开启时，每次运行的日志写入
|
||||
`logs/<log_prefix>_<timestamp>/`（包含一个 `runner.log` 及每个 worker 进程的
|
||||
`worker-<pid>.log`）。
|
||||
|
||||
## 6. 终止运行
|
||||
|
||||
并行运行会派生进程树。若要干净地终止某次运行及其全部 worker：
|
||||
|
||||
```bash
|
||||
bash benchmark/kill.sh <PID>
|
||||
```
|
||||
|
||||
该脚本会先向整个进程树发送 `SIGTERM` 优雅终止，对 5 秒内未退出的进程再升级为 `SIGKILL`。
|
||||
|
||||
## 7. 参考结果
|
||||
|
||||
已记录的评测结果见：
|
||||
|
||||
- [`longmemeval.md`](./results_md/longmemeval.md)
|
||||
- [`beam.md`](./results_md/beam.md)
|
||||
124
benchmark/beam/README.md
Normal file
124
benchmark/beam/README.md
Normal file
|
|
@ -0,0 +1,124 @@
|
|||
[中文版 / Chinese version](./README_ZH.md)
|
||||
|
||||
# BEAM Benchmark
|
||||
|
||||
BEAM is a benchmark for **memory capability over long-context chat cases**. Each
|
||||
case contains a very long chat history split into batches; ReMe converts each
|
||||
batch into a session, ingests them in chronological order, then answers probing
|
||||
questions via an agentic (ReAct) mode. Answers are scored with BEAM's
|
||||
rubric-based `answer_judge` job, which produces both a graded score and a binary
|
||||
verdict, and per-type averages are reported.
|
||||
|
||||
BEAM ships dataset variants by chat size — `100K` / `500K` / `1M` / `10M` — so
|
||||
memory systems can be stressed at different context lengths. Question types
|
||||
include abstention, contradiction resolution, event ordering, information
|
||||
extraction, instruction following, knowledge update, multi-session reasoning,
|
||||
preference following, summarization, and temporal reasoning.
|
||||
|
||||
> For the shared setup (dependencies, credentials, log conventions) see the
|
||||
> [top-level benchmark README](../README.md).
|
||||
|
||||
## 1. Get the Dataset
|
||||
|
||||
BEAM is a public repository, cloned into `benchmark/beam/dataset/`:
|
||||
|
||||
```bash
|
||||
mkdir -p benchmark/beam/dataset
|
||||
cd benchmark/beam/dataset
|
||||
git clone https://github.com/mohammadtavakoli78/BEAM.git
|
||||
```
|
||||
|
||||
After cloning, `benchmark/beam/dataset/BEAM/` should contain `chats/`, `src/`,
|
||||
`topics/` and other subdirectories.
|
||||
|
||||
## 2. Run
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
python benchmark/beam/run.py
|
||||
python benchmark/beam/run.py --config benchmark/beam/config.yaml
|
||||
python benchmark/beam/run.py -q # quiet
|
||||
python benchmark/beam/run.py --eval_only # reuse existing workspaces, query + judge only
|
||||
```
|
||||
|
||||
## 3. Pipeline
|
||||
|
||||
1. For each case, load `chat.json` and convert each batch into a ReMe session.
|
||||
2. Ingest sessions in chronological order into an isolated workspace, then `digest_update`.
|
||||
3. Answer each probing question via agentic (ReAct) mode.
|
||||
4. Score answers with BEAM's rubric-based `answer_judge` job and print per-type averages.
|
||||
|
||||
## 4. Key config — `benchmark/beam/config.yaml`
|
||||
|
||||
| Key | Meaning |
|
||||
| --- | --- |
|
||||
| `dataset.beam_root` | BEAM dataset root (`benchmark/beam/dataset/BEAM`). |
|
||||
| `dataset.chat_size` | Variant to run: `100K` / `500K` / `1M` / `10M`. |
|
||||
| `dataset.case_ids` | Specific cases (e.g. `["1","2"]`); empty = all cases. |
|
||||
| `dataset.start_index` / `num_items` | Case pagination (`num_items` `0` = all). |
|
||||
| `dataset.workspace_root` | Per-case workspace root (`benchmark/beam/workspaces/beam`). |
|
||||
| `evaluation.num_workers` | `0` = auto, `1` = sequential, `>1` = parallel. |
|
||||
| `reme.config` | ReMe config used (`beam.yaml`). |
|
||||
| `output.dir` | Results directory (`benchmark/beam/results`). |
|
||||
|
||||
## 5. Outputs
|
||||
|
||||
Results are JSON files written to `output.dir` as
|
||||
`results_<chat_size>_<timestamp>.json`, with a per-type score summary also
|
||||
printed to the console. Logging conventions are shared across benchmarks — see
|
||||
the [top-level README](../README.md#outputs--logs).
|
||||
|
||||
## 6. Reference Results
|
||||
|
||||
> The results below use the longmemeval-version prompt.
|
||||
|
||||
### 100K
|
||||
|
||||
agentscope==2.0.4.post1, conda reme env, 20 workers, eval-only (reusing prebuilt memory)
|
||||
(2026-08-05, 20 cases / 400 Qs, total 46.0 min)
|
||||
|
||||
| Type | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|
||||
|---|---|---|---|---|---|---|
|
||||
| abstention | 0.550 | 0.550 | 96,031 | 1,070 | 97,101 | 4.58 |
|
||||
| contradiction_resolution | 0.438 | 0.412 | 32,263 | 872 | 33,135 | 2.48 |
|
||||
| event_ordering | 0.501 | 0.423 | 140,195 | 5,163 | 145,358 | 4.70 |
|
||||
| information_extraction | 0.873 | 0.832 | 50,245 | 883 | 51,128 | 3.15 |
|
||||
| instruction_following | 0.750 | 0.725 | 37,986 | 848 | 38,834 | 2.67 |
|
||||
| knowledge_update | 0.688 | 0.675 | 31,198 | 651 | 31,849 | 2.27 |
|
||||
| multi_session_reasoning | 0.626 | 0.584 | 85,038 | 4,563 | 89,601 | 4.28 |
|
||||
| preference_following | 0.925 | 0.912 | 34,281 | 989 | 35,270 | 2.50 |
|
||||
| summarization | 0.623 | 0.461 | 89,657 | 2,056 | 91,713 | 4.12 |
|
||||
| temporal_reasoning | 0.637 | 0.625 | 34,563 | 1,049 | 35,612 | 2.52 |
|
||||
| **OVERALL** | **0.661** | **0.620** | **63,146** | **1,814** | **64,960** | **3.33** |
|
||||
|
||||
Memory Construction average token consumption (default agent, full build over 20 cases):
|
||||
|
||||
| Agent | input tok/case | output tok/case | total tok/case |
|
||||
|---|---|---|---|
|
||||
| default | 2,172,316 | 136,697 | 2,309,013 |
|
||||
|
||||
### 1M
|
||||
|
||||
agentscope==2.0.4.post1, conda reme env, 20 workers, full memory build
|
||||
(2026-08-05, 35 cases / 700 Qs, total 459.2 min)
|
||||
|
||||
| Type | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|
||||
|---|---|---|---|---|---|---|
|
||||
| abstention | 0.429 | 0.429 | 118,707 | 1,178 | 119,886 | 4.20 |
|
||||
| contradiction_resolution | 0.391 | 0.364 | 49,787 | 810 | 50,597 | 2.50 |
|
||||
| event_ordering | 0.558 | 0.456 | 201,514 | 3,889 | 205,403 | 4.79 |
|
||||
| information_extraction | 0.809 | 0.772 | 78,950 | 894 | 79,844 | 3.00 |
|
||||
| instruction_following | 0.852 | 0.832 | 55,757 | 924 | 56,681 | 2.81 |
|
||||
| knowledge_update | 0.779 | 0.771 | 45,981 | 665 | 46,646 | 2.37 |
|
||||
| multi_session_reasoning | 0.658 | 0.612 | 138,133 | 2,873 | 141,006 | 4.40 |
|
||||
| preference_following | 0.798 | 0.777 | 51,796 | 920 | 52,716 | 2.53 |
|
||||
| summarization | 0.693 | 0.537 | 158,794 | 2,905 | 161,700 | 4.44 |
|
||||
| temporal_reasoning | 0.536 | 0.536 | 100,176 | 3,148 | 103,324 | 3.90 |
|
||||
| **OVERALL** | **0.650** | **0.609** | **99,959** | **1,821** | **101,780** | **3.49** |
|
||||
|
||||
Memory Construction average token consumption (default agent, full build over 35 cases):
|
||||
|
||||
| Agent | input tok/case | output tok/case | total tok/case |
|
||||
|---|---|---|---|
|
||||
| default | 31,943,817 | 1,417,061 | 33,360,878 |
|
||||
119
benchmark/beam/README_ZH.md
Normal file
119
benchmark/beam/README_ZH.md
Normal file
|
|
@ -0,0 +1,119 @@
|
|||
# BEAM 评测
|
||||
|
||||
[English version](./README.md)
|
||||
|
||||
BEAM 是一个面向**长上下文对话场景**的记忆能力评测基准。每个 case 包含一段被切分为多个
|
||||
batch 的超长对话；ReMe 将每个 batch 转换为一个会话，按时间顺序摄入后，以 agentic（ReAct）
|
||||
模式回答探测问题。答案由 BEAM 基于 rubric 的 `answer_judge` 任务打分，同时给出分级分数与二元
|
||||
判定，并输出各类型平均分。
|
||||
|
||||
BEAM 按对话规模提供多种数据变体 —— `100K` / `500K` / `1M` / `10M`，可在不同上下文长度下
|
||||
压测记忆系统。题型包括 abstention（拒答）、contradiction resolution（矛盾消解）、event
|
||||
ordering（事件排序）、information extraction（信息抽取）、instruction following（指令遵循）、
|
||||
knowledge update（知识更新）、multi-session reasoning（多会话推理）、preference following
|
||||
（偏好遵循）、summarization（摘要）与 temporal reasoning（时间推理）。
|
||||
|
||||
> 公共设置（依赖、凭据、日志约定）见[总评测说明](../README_ZH.md)。
|
||||
|
||||
## 1. 获取数据集
|
||||
|
||||
BEAM 是公开仓库，clone 到 `benchmark/beam/dataset/` 下：
|
||||
|
||||
```bash
|
||||
mkdir -p benchmark/beam/dataset
|
||||
cd benchmark/beam/dataset
|
||||
git clone https://github.com/mohammadtavakoli78/BEAM.git
|
||||
```
|
||||
|
||||
clone 完成后，`benchmark/beam/dataset/BEAM/` 目录下应包含 `chats/`、`src/`、`topics/` 等子目录。
|
||||
|
||||
## 2. 运行
|
||||
|
||||
在仓库根目录执行：
|
||||
|
||||
```bash
|
||||
python benchmark/beam/run.py
|
||||
python benchmark/beam/run.py --config benchmark/beam/config.yaml
|
||||
python benchmark/beam/run.py -q # 安静模式
|
||||
python benchmark/beam/run.py --eval_only # 复用已有工作区，仅执行查询 + 评判
|
||||
```
|
||||
|
||||
## 3. 流程
|
||||
|
||||
1. 为每个 case 加载 `chat.json`，将每个 batch 转换为一个 ReMe 会话。
|
||||
2. 按时间顺序将会话摄入独立工作区，随后执行 `digest_update`。
|
||||
3. 以 agentic（ReAct）模式回答每个探测问题。
|
||||
4. 通过 BEAM 基于 rubric 的 `answer_judge` 任务打分，并输出各类型平均分。
|
||||
|
||||
## 4. 关键配置 —— `benchmark/beam/config.yaml`
|
||||
|
||||
| 配置项 | 含义 |
|
||||
| --- | --- |
|
||||
| `dataset.beam_root` | BEAM 数据集根目录（`benchmark/beam/dataset/BEAM`）。 |
|
||||
| `dataset.chat_size` | 运行的变体：`100K` / `500K` / `1M` / `10M`。 |
|
||||
| `dataset.case_ids` | 指定 case（如 `["1","2"]`），空表示全部。 |
|
||||
| `dataset.start_index` / `num_items` | case 分页（`num_items` 为 `0` 表示全部）。 |
|
||||
| `dataset.workspace_root` | case 工作区根目录（`benchmark/beam/workspaces/beam`）。 |
|
||||
| `evaluation.num_workers` | `0` = 自动，`1` = 串行，`>1` = 并行。 |
|
||||
| `reme.config` | 使用的 ReMe 配置（`beam.yaml`）。 |
|
||||
| `output.dir` | 结果目录（`benchmark/beam/results`）。 |
|
||||
|
||||
## 5. 输出
|
||||
|
||||
结果以 JSON 文件写入 `output.dir`，文件名为 `results_<chat_size>_<timestamp>.json`，
|
||||
同时控制台会打印含各类型分数的汇总。日志约定在各基准间通用，见
|
||||
[总说明](../README_ZH.md#输出与日志)。
|
||||
|
||||
## 6. 参考结果
|
||||
|
||||
> 以下结果使用 longmemeval 版本的 prompt。
|
||||
|
||||
### 100K
|
||||
|
||||
agentscope==2.0.4.post1，conda reme 环境，20 并发，eval-only（复用已构建 memory）
|
||||
（2026-08-05，20 cases / 400 Qs，总耗时 46.0 min）
|
||||
|
||||
| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|
||||
|---|---|---|---|---|---|---|
|
||||
| abstention | 0.550 | 0.550 | 96,031 | 1,070 | 97,101 | 4.58 |
|
||||
| contradiction_resolution | 0.438 | 0.412 | 32,263 | 872 | 33,135 | 2.48 |
|
||||
| event_ordering | 0.501 | 0.423 | 140,195 | 5,163 | 145,358 | 4.70 |
|
||||
| information_extraction | 0.873 | 0.832 | 50,245 | 883 | 51,128 | 3.15 |
|
||||
| instruction_following | 0.750 | 0.725 | 37,986 | 848 | 38,834 | 2.67 |
|
||||
| knowledge_update | 0.688 | 0.675 | 31,198 | 651 | 31,849 | 2.27 |
|
||||
| multi_session_reasoning | 0.626 | 0.584 | 85,038 | 4,563 | 89,601 | 4.28 |
|
||||
| preference_following | 0.925 | 0.912 | 34,281 | 989 | 35,270 | 2.50 |
|
||||
| summarization | 0.623 | 0.461 | 89,657 | 2,056 | 91,713 | 4.12 |
|
||||
| temporal_reasoning | 0.637 | 0.625 | 34,563 | 1,049 | 35,612 | 2.52 |
|
||||
| **OVERALL** | **0.661** | **0.620** | **63,146** | **1,814** | **64,960** | **3.33** |
|
||||
|
||||
Memory Construction 平均 token 消耗（default agent，20 cases 全量构建）：
|
||||
|
||||
| Agent | input tok/case | output tok/case | total tok/case |
|
||||
|---|---|---|---|
|
||||
| default | 2,172,316 | 136,697 | 2,309,013 |
|
||||
|
||||
### 1M
|
||||
|
||||
agentscope==2.0.4.post1，conda reme 环境，20 并发，全量构建 memory
|
||||
（2026-08-05，35 cases / 700 Qs，总耗时 459.2 min）
|
||||
|
||||
| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|
||||
|---|---|---|---|---|---|---|
|
||||
| abstention | 0.429 | 0.429 | 118,707 | 1,178 | 119,886 | 4.20 |
|
||||
| contradiction_resolution | 0.391 | 0.364 | 49,787 | 810 | 50,597 | 2.50 |
|
||||
| event_ordering | 0.558 | 0.456 | 201,514 | 3,889 | 205,403 | 4.79 |
|
||||
| information_extraction | 0.809 | 0.772 | 78,950 | 894 | 79,844 | 3.00 |
|
||||
| instruction_following | 0.852 | 0.832 | 55,757 | 924 | 56,681 | 2.81 |
|
||||
| knowledge_update | 0.779 | 0.771 | 45,981 | 665 | 46,646 | 2.37 |
|
||||
| multi_session_reasoning | 0.658 | 0.612 | 138,133 | 2,873 | 141,006 | 4.40 |
|
||||
| preference_following | 0.798 | 0.777 | 51,796 | 920 | 52,716 | 2.53 |
|
||||
| summarization | 0.693 | 0.537 | 158,794 | 2,905 | 161,700 | 4.44 |
|
||||
| temporal_reasoning | 0.536 | 0.536 | 100,176 | 3,148 | 103,324 | 3.90 |
|
||||
| **OVERALL** | **0.650** | **0.609** | **99,959** | **1,821** | **101,780** | **3.49** |
|
||||
|
||||
Memory Construction 平均 token 消耗（default agent，35 cases 全量构建）：
|
||||
|
||||
| Agent | input tok/case | output tok/case | total tok/case |
|
||||
|---|---|---|---|
|
||||
| default | 31,943,817 | 1,417,061 | 33,360,878 |
|
||||
96
benchmark/longmemeval/README.md
Normal file
96
benchmark/longmemeval/README.md
Normal file
|
|
@ -0,0 +1,96 @@
|
|||
[中文版 / Chinese version](./README_ZH.md)
|
||||
|
||||
# LongMemEval Benchmark
|
||||
|
||||
LongMemEval is a benchmark for **long-term memory over multi-session chat
|
||||
histories**. Each item provides a chronologically ordered set of chat sessions
|
||||
between a user and an assistant, followed by a probing question whose answer is
|
||||
only recoverable by reasoning over the user-owned memory. ReMe ingests the
|
||||
sessions into an isolated per-item workspace, answers the question via an
|
||||
agentic (ReAct) mode, and scores the answer with an LLM-as-judge.
|
||||
|
||||
Question types include single-session (user / assistant / preference),
|
||||
multi-session reasoning, knowledge update, and temporal reasoning.
|
||||
|
||||
> For the shared setup (dependencies, credentials, log conventions) see the
|
||||
> [top-level benchmark README](../README.md).
|
||||
|
||||
## 1. Get the Dataset
|
||||
|
||||
ReMe uses only the **cleaned-S** split, hosted on HuggingFace:
|
||||
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2).
|
||||
The download script fetches it via the hf-mirror.com mirror; to use a different
|
||||
mirror, modify `BASE_URL` in [`download.py`](./download.py).
|
||||
|
||||
```bash
|
||||
cd benchmark/longmemeval
|
||||
python download.py # saves dataset/longmemeval_s_reme_cleaned.json; skips if already present
|
||||
```
|
||||
|
||||
Ground truth is embedded in the data file.
|
||||
|
||||
## 2. Run
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
python benchmark/longmemeval/run.py
|
||||
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
|
||||
python benchmark/longmemeval/run.py -q # quiet: only eval-level logs
|
||||
python benchmark/longmemeval/run.py --log-level WARNING # reduce eval runner logs
|
||||
python benchmark/longmemeval/run.py --reme-log-level WARNING # reduce reme internal logs
|
||||
python benchmark/longmemeval/run.py --eval_only # reuse existing workspaces, query + judge only
|
||||
```
|
||||
|
||||
## 3. Pipeline
|
||||
|
||||
1. Load the dataset (ground truth is embedded in the data file).
|
||||
2. For each item, create an isolated workspace and ingest sessions in chronological order.
|
||||
3. Trigger `auto_dream` when consecutive sessions cross the configured hour (default 23:00).
|
||||
4. Answer each question via agentic (ReAct) mode.
|
||||
5. Judge the answer (binary yes/no) with the `answer_judge` job and print per-type accuracy.
|
||||
|
||||
## 4. Key config — `benchmark/longmemeval/config.yaml`
|
||||
|
||||
| Key | Meaning |
|
||||
| --- | --- |
|
||||
| `dataset.path` | Dataset file to evaluate (e.g. `longmemeval_s_reme_cleaned.json`); ground truth is included. |
|
||||
| `dataset.start_index` / `num_items` | Slice of items to evaluate. |
|
||||
| `dataset.question_types` | Filter by question type; empty = all. |
|
||||
| `dataset.workspace_root` | Per-item workspace root (`benchmark/longmemeval/workspaces/longmemeval-s`). |
|
||||
| `evaluation.num_workers` | `0` = auto (cpu-2), `1` = sequential, `>1` = parallel. |
|
||||
| `evaluation.filter_future_sessions` | Only ingest sessions with timestamp ≤ `question_date`. |
|
||||
| `reme.config` | ReMe config used (`lme.yaml`). |
|
||||
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | Dream triggering behavior. |
|
||||
| `output.dir` | Results directory (`benchmark/longmemeval/results`). |
|
||||
|
||||
## 5. Outputs
|
||||
|
||||
Results are JSON files written to `output.dir` as `results_<timestamp>.json`,
|
||||
with a per-type accuracy summary also printed to the console. Logging
|
||||
conventions are shared across benchmarks — see the
|
||||
[top-level README](../README.md#outputs--logs).
|
||||
|
||||
## 6. Reference Results
|
||||
|
||||
### cleaned-s
|
||||
|
||||
**Basic settings**
|
||||
|
||||
1. Modified auto-memory prompt, auto-dream disabled.
|
||||
2. All sessions in reme-memory are strictly earlier than the question time.
|
||||
|
||||
**Results**
|
||||
|
||||
agentscope==2.0.4.post1, conda reme env, 32 workers, eval-only (reusing prebuilt memory)
|
||||
(2026-08-06, 500 items, total 10.0 min)
|
||||
|
||||
| Type | Agentic | input tok/q | output tok/q | total tok/q | tool calls/q |
|
||||
|---|---|---|---|---|---|
|
||||
| knowledge-update | 0.910 | 31,581 | 589 | 32,169 | 2.90 |
|
||||
| multi-session | 0.842 | 52,837 | 1,474 | 54,311 | 4.21 |
|
||||
| single-session-assistant | 1.000 | 15,596 | 279 | 15,875 | 1.89 |
|
||||
| single-session-preference | 0.633 | 36,802 | 818 | 37,620 | 3.60 |
|
||||
| single-session-user | 0.986 | 27,433 | 359 | 27,792 | 2.60 |
|
||||
| temporal-reasoning | 0.902 | 62,674 | 985 | 63,659 | 4.97 |
|
||||
| **OVERALL** | **0.894** | **43,448** | **876** | **44,324** | **3.69** |
|
||||
90
benchmark/longmemeval/README_ZH.md
Normal file
90
benchmark/longmemeval/README_ZH.md
Normal file
|
|
@ -0,0 +1,90 @@
|
|||
# LongMemEval 评测
|
||||
|
||||
[English version](./README.md)
|
||||
|
||||
LongMemEval 是一个面向**多轮多会话历史的长期记忆能力**的评测基准。每个条目提供一组按时间
|
||||
顺序排列的用户与助手之间的会话，以及一个只能通过推理用户自有记忆才能回答的探测问题。ReMe
|
||||
将会话摄入按条目隔离的工作区，以 agentic（ReAct）模式回答问题，最后由 LLM-as-judge 打分。
|
||||
|
||||
题型包括单会话（user / assistant / preference）、多会话推理、知识更新与时间推理等。
|
||||
|
||||
> 公共设置（依赖、凭据、日志约定）见[总评测说明](../README_ZH.md)。
|
||||
|
||||
## 1. 获取数据集
|
||||
|
||||
ReMe 仅使用 **cleaned-S** 版本，数据托管在 HuggingFace：
|
||||
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)。
|
||||
下载脚本经 hf-mirror.com 镜像源获取，如需更换源请修改 [`download.py`](./download.py) 中的
|
||||
`BASE_URL`。
|
||||
|
||||
```bash
|
||||
cd benchmark/longmemeval
|
||||
python download.py # 保存为 dataset/longmemeval_s_reme_cleaned.json，已存在则自动跳过
|
||||
```
|
||||
|
||||
ground truth 已内嵌在数据文件中。
|
||||
|
||||
## 2. 运行
|
||||
|
||||
在仓库根目录执行：
|
||||
|
||||
```bash
|
||||
python benchmark/longmemeval/run.py
|
||||
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
|
||||
python benchmark/longmemeval/run.py -q # 安静模式：仅评测级日志
|
||||
python benchmark/longmemeval/run.py --log-level WARNING # 降低评测 runner 日志
|
||||
python benchmark/longmemeval/run.py --reme-log-level WARNING # 降低 reme 内部日志
|
||||
python benchmark/longmemeval/run.py --eval_only # 复用已有工作区，仅执行查询 + 评判
|
||||
```
|
||||
|
||||
## 3. 流程
|
||||
|
||||
1. 加载数据集（ground truth 已内嵌在数据文件中）。
|
||||
2. 为每个条目创建独立工作区，按时间顺序摄入会话。
|
||||
3. 当相邻会话跨越配置的时刻（默认 23:00）时触发 `auto_dream`。
|
||||
4. 以 agentic（ReAct）模式回答每个问题。
|
||||
5. 通过 `answer_judge` 任务对答案做二元（yes/no）评判，并输出各类型准确率。
|
||||
|
||||
## 4. 关键配置 —— `benchmark/longmemeval/config.yaml`
|
||||
|
||||
| 配置项 | 含义 |
|
||||
| --- | --- |
|
||||
| `dataset.path` | 待评测的数据集文件（如 `longmemeval_s_reme_cleaned.json`），已包含 ground truth。 |
|
||||
| `dataset.start_index` / `num_items` | 评测条目的切片范围。 |
|
||||
| `dataset.question_types` | 按问题类型过滤，空表示全部。 |
|
||||
| `dataset.workspace_root` | 条目工作区根目录（`benchmark/longmemeval/workspaces/longmemeval-s`）。 |
|
||||
| `evaluation.num_workers` | `0` = 自动（cpu-2），`1` = 串行，`>1` = 并行。 |
|
||||
| `evaluation.filter_future_sessions` | 仅摄入时间戳 ≤ `question_date` 的会话。 |
|
||||
| `reme.config` | 使用的 ReMe 配置（`lme.yaml`）。 |
|
||||
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | dream 触发行为。 |
|
||||
| `output.dir` | 结果目录（`benchmark/longmemeval/results`）。 |
|
||||
|
||||
## 5. 输出
|
||||
|
||||
结果以 JSON 文件写入 `output.dir`，文件名为 `results_<timestamp>.json`，
|
||||
同时控制台会打印含各类型准确率的汇总。日志约定在各基准间通用，见
|
||||
[总说明](../README_ZH.md#输出与日志)。
|
||||
|
||||
## 6. 参考结果
|
||||
|
||||
### cleaned-s
|
||||
|
||||
**基础设置**
|
||||
|
||||
1. 使用修改后的 auto-memory prompt，关闭 auto-dream 机制
|
||||
2. reme-memory 中的全部 session 的时间一定早于 question 的时间
|
||||
|
||||
**结果**
|
||||
|
||||
agentscope==2.0.4.post1, conda reme env, 32 workers, eval-only（复用预构建记忆）
|
||||
（2026-08-06，500 题，总计 10.0 min）
|
||||
|
||||
| 类型 | Agentic | input tok/q | output tok/q | total tok/q | tool calls/q |
|
||||
|---|---|---|---|---|---|
|
||||
| knowledge-update | 0.910 | 31,581 | 589 | 32,169 | 2.90 |
|
||||
| multi-session | 0.842 | 52,837 | 1,474 | 54,311 | 4.21 |
|
||||
| single-session-assistant | 1.000 | 15,596 | 279 | 15,875 | 1.89 |
|
||||
| single-session-preference | 0.633 | 36,802 | 818 | 37,620 | 3.60 |
|
||||
| single-session-user | 0.986 | 27,433 | 359 | 27,792 | 2.60 |
|
||||
| temporal-reasoning | 0.902 | 62,674 | 985 | 63,659 | 4.97 |
|
||||
| **OVERALL** | **0.894** | **43,448** | **876** | **44,324** | **3.69** |
|
||||
76
benchmark/longmemeval/kill.sh
Normal file
76
benchmark/longmemeval/kill.sh
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
#!/bin/bash
|
||||
# 杀死指定进程及其所有子进程
|
||||
# Usage: bash kill.sh <PID>
|
||||
|
||||
if [ -z "$1" ]; then
|
||||
echo "Usage: bash kill.sh <PID>"
|
||||
echo " 杀死指定进程及其所有子进程"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
PID=$1
|
||||
|
||||
# 检查进程是否存在
|
||||
if ! kill -0 "$PID" 2>/dev/null; then
|
||||
echo "进程 $PID 不存在"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 递归收集所有子进程（包括子进程的子进程）
|
||||
collect_children() {
|
||||
local parent=$1
|
||||
local children
|
||||
children=$(ps -o pid= --ppid "$parent" 2>/dev/null | tr -d ' ')
|
||||
for child in $children; do
|
||||
collect_children "$child"
|
||||
done
|
||||
echo "$parent"
|
||||
}
|
||||
|
||||
# 收集进程树（子进程在前，父进程在后，保证先杀子再杀父）
|
||||
PROCESS_TREE=$(collect_children "$PID")
|
||||
TOTAL=$(echo "$PROCESS_TREE" | wc -l | tr -d ' ')
|
||||
|
||||
echo "进程树（共 $TOTAL 个进程）:"
|
||||
while read -r p; do
|
||||
cmd=$(ps -o args= -p "$p" 2>/dev/null | head -c 80)
|
||||
printf " PID=%-8s %s\n" "$p" "$cmd"
|
||||
done <<< "$PROCESS_TREE"
|
||||
|
||||
# 先 SIGTERM 优雅终止
|
||||
echo ""
|
||||
echo "发送 SIGTERM..."
|
||||
while read -r p; do
|
||||
kill "$p" 2>/dev/null
|
||||
done <<< "$PROCESS_TREE"
|
||||
|
||||
# 等待最多 5 秒
|
||||
for i in $(seq 1 5); do
|
||||
alive=false
|
||||
while read -r p; do
|
||||
if kill -0 "$p" 2>/dev/null; then
|
||||
alive=true
|
||||
fi
|
||||
done <<< "$PROCESS_TREE"
|
||||
if [ "$alive" = false ]; then
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
# 检查是否还有残留，强制 SIGKILL
|
||||
remaining=false
|
||||
while read -r p; do
|
||||
if kill -0 "$p" 2>/dev/null; then
|
||||
remaining=true
|
||||
fi
|
||||
done <<< "$PROCESS_TREE"
|
||||
|
||||
if [ "$remaining" = true ]; then
|
||||
echo "部分进程未响应，发送 SIGKILL..."
|
||||
while read -r p; do
|
||||
kill -9 "$p" 2>/dev/null
|
||||
done <<< "$PROCESS_TREE"
|
||||
fi
|
||||
|
||||
echo "已终止进程树（根 PID=$PID，共 $TOTAL 个进程）"
|
||||
|
|
@ -1,54 +0,0 @@
|
|||
# beam result
|
||||
|
||||
## longmemeval版本的prompt
|
||||
|
||||
### 100K
|
||||
|
||||
|
||||
agentscope==2.0.4.post1, conda reme 环境, 20 并发, eval-only（复用已构建 memory）
|
||||
(2026-08-05, 20 cases / 400 Qs, 总耗时 46.0 min)
|
||||
|
||||
| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|
||||
|---|---|---|---|---|---|---|
|
||||
| abstention | 0.550 | 0.550 | 96,031 | 1,070 | 97,101 | 4.58 |
|
||||
| contradiction_resolution | 0.438 | 0.412 | 32,263 | 872 | 33,135 | 2.48 |
|
||||
| event_ordering | 0.501 | 0.423 | 140,195 | 5,163 | 145,358 | 4.70 |
|
||||
| information_extraction | 0.873 | 0.832 | 50,245 | 883 | 51,128 | 3.15 |
|
||||
| instruction_following | 0.750 | 0.725 | 37,986 | 848 | 38,834 | 2.67 |
|
||||
| knowledge_update | 0.688 | 0.675 | 31,198 | 651 | 31,849 | 2.27 |
|
||||
| multi_session_reasoning | 0.626 | 0.584 | 85,038 | 4,563 | 89,601 | 4.28 |
|
||||
| preference_following | 0.925 | 0.912 | 34,281 | 989 | 35,270 | 2.50 |
|
||||
| summarization | 0.623 | 0.461 | 89,657 | 2,056 | 91,713 | 4.12 |
|
||||
| temporal_reasoning | 0.637 | 0.625 | 34,563 | 1,049 | 35,612 | 2.52 |
|
||||
| **OVERALL** | **0.661** | **0.620** | **63,146** | **1,814** | **64,960** | **3.33** |
|
||||
|
||||
Memory Construction 平均 token 消耗（default agent, 20 cases 全量构建）:
|
||||
|
||||
| Agent | input tok/case | output tok/case | total tok/case |
|
||||
|---|---|---|---|
|
||||
| default | 2,172,316 | 136,697 | 2,309,013 |
|
||||
|
||||
### 1M
|
||||
|
||||
agentscope==2.0.4.post1, conda reme 环境, 20 并发, 全量构建 memory
|
||||
(2026-08-05, 35 cases / 700 Qs, 总耗时 459.2 min)
|
||||
|
||||
| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|
||||
|---|---|---|---|---|---|---|
|
||||
| abstention | 0.429 | 0.429 | 118,707 | 1,178 | 119,886 | 4.20 |
|
||||
| contradiction_resolution | 0.391 | 0.364 | 49,787 | 810 | 50,597 | 2.50 |
|
||||
| event_ordering | 0.558 | 0.456 | 201,514 | 3,889 | 205,403 | 4.79 |
|
||||
| information_extraction | 0.809 | 0.772 | 78,950 | 894 | 79,844 | 3.00 |
|
||||
| instruction_following | 0.852 | 0.832 | 55,757 | 924 | 56,681 | 2.81 |
|
||||
| knowledge_update | 0.779 | 0.771 | 45,981 | 665 | 46,646 | 2.37 |
|
||||
| multi_session_reasoning | 0.658 | 0.612 | 138,133 | 2,873 | 141,006 | 4.40 |
|
||||
| preference_following | 0.798 | 0.777 | 51,796 | 920 | 52,716 | 2.53 |
|
||||
| summarization | 0.693 | 0.537 | 158,794 | 2,905 | 161,700 | 4.44 |
|
||||
| temporal_reasoning | 0.536 | 0.536 | 100,176 | 3,148 | 103,324 | 3.90 |
|
||||
| **OVERALL** | **0.650** | **0.609** | **99,959** | **1,821** | **101,780** | **3.49** |
|
||||
|
||||
Memory Construction 平均 token 消耗（default agent, 35 cases 全量构建）:
|
||||
|
||||
| Agent | input tok/case | output tok/case | total tok/case |
|
||||
|---|---|---|---|
|
||||
| default | 31,943,817 | 1,417,061 | 33,360,878 |
|
||||
|
|
@ -1,92 +0,0 @@
|
|||
# LongMemEval 数据集测试结果
|
||||
|
||||
## cleaned-s
|
||||
|
||||
**basic settings**
|
||||
|
||||
1. 使用修改后的auto-memory prompt，关闭auto-dream机制
|
||||
2. reme-memory中的全部session的时间一定早于question的时间
|
||||
|
||||
**results **
|
||||
|
||||
1. Agentic answer框架回答，每次最多调用5次search
|
||||
|
||||
| Category | Total | Correct | Wrong | Accuracy |
|
||||
|---|---|---|---|---|
|
||||
| single-session-user | 70 | 66 | 4 | 94.3% |
|
||||
| single-session-assistant | 56 | 52 | 4 | 92.9% |
|
||||
| knowledge-update | 78 | 60 | 18 | 76.9% |
|
||||
| multi-session | 133 | 93 | 40 | 69.9% |
|
||||
| temporal-reasoning | 133 | 78 | 55 | 58.6% |
|
||||
| single-session-preference | 30 | 8 | 22 | 26.7% |
|
||||
| **Overall** | **500** | **357** | **143** | **71.4%** |
|
||||
|
||||
2. prompted-based amswer，每次固定使用原始query召回10个fileChunk
|
||||
|
||||
| Category | Total | Correct | Wrong | Accuracy |
|
||||
|---|---|---|---|---|
|
||||
| single-session-assistant | 56 | 56 | 0 | 100.0% |
|
||||
| single-session-user | 70 | 67 | 3 | 95.7% |
|
||||
| knowledge-update | 78 | 69 | 9 | 88.5% |
|
||||
| multi-session | 133 | 99 | 34 | 74.4% |
|
||||
| temporal-reasoning | 133 | 83 | 50 | 62.4% |
|
||||
| single-session-preference | 30 | 16 | 14 | 53.3% |
|
||||
| **Overall** | **500** | **390** | **110** | **78.0%** |
|
||||
|
||||
3. golden session。 使用与prompt-based answer相似的方法，唯一区别是，输入的chunk是longMemEval提供的golden session。
|
||||
|
||||
| Category | Total | Correct | Wrong | Accuracy |
|
||||
|---|---|---|---|---|
|
||||
| single-session-assistant | 56 | 56 | 0 | 100.0% |
|
||||
| single-session-user | 70 | 69 | 1 | 98.6% |
|
||||
| knowledge-update | 78 | 74 | 4 | 94.9% |
|
||||
| temporal-reasoning | 133 | 124 | 9 | 93.2% |
|
||||
| multi-session | 133 | 117 | 16 | 88.0% |
|
||||
| single-session-preference | 30 | 17 | 13 | 56.7% |
|
||||
| **Overall** | **500** | **457** | **43** | **91.4%** |
|
||||
|
||||
4. golden session + time filter. 和上面一个实验的区别是，输入的golden被过滤了一次，要求输入session的时间戳必须早于question的时间才行。
|
||||
|
||||
一共被过滤掉了75个session，44个question受到了影响。temperal reasoning类型受影响最大。有20个case不包含任何一个groundtruth session。 根据golden session回答正确并且golden session非空，一共有424个case。
|
||||
|
||||
| Category | Total | Correct | Wrong | Accuracy |
|
||||
|---|---|---|---|---|
|
||||
| knowledge-update | 78 | 75 | 3 | 96.2% |
|
||||
| single-session-user | 70 | 67 | 3 | 95.7% |
|
||||
| multi-session | 133 | 122 | 11 | 91.7% |
|
||||
| single-session-assistant | 56 | 55 | 1 | 98.2% |
|
||||
| temporal-reasoning | 133 | 91 | 42 | 68.4% |
|
||||
| single-session-preference | 30 | 16 | 14 | 53.3% |
|
||||
| **Overall** | **500** | **426** | **74** | **85.2%** |
|
||||
|
||||
5. 关闭auto-memory机制，根据原始query一次性混合检索召回原始session，计算recall.
|
||||
|
||||
| Category | Total | yes-judge | recall@5 / yes | recall@10 / yes |
|
||||
|---|---|---|---|---|
|
||||
| knowledge-update | 78 | 75 | 99.3% | 100% |
|
||||
| single-session-user | 70 | 67 | 100% | 100% |
|
||||
| multi-session | 133 | 122 | 91.8% | 95.8% |
|
||||
| single-session-assistant | 56 | 55 | 100% | 100% |
|
||||
| temporal-reasoning | 133 | 91 | 87.6% | 94.2% |
|
||||
| single-session-preference | 30 | 16 | 100% | 100% |
|
||||
| **Overall** | **500** | **426** | **87.6%** | **94.2%** |
|
||||
|
||||
|
||||
## 最终groundtruth
|
||||
|
||||
### agentic + prompted（最终GT，2026-07-16）
|
||||
|
||||
|
||||
| Category | Total | Agentic | Prompted limit=15 |
|
||||
|---|---|---|---|
|
||||
| single-session-assistant | 56 | 56/56 (100.0%) | 54/56 (96.4%) |
|
||||
| single-session-user | 70 | 66/70 (94.3%) | 62/70 (88.6%) |
|
||||
| knowledge-update | 78 | 75/78 (96.2%) | 67/78 (85.9%) |
|
||||
| temporal-reasoning | 133 | 122/133 (91.7%) | 117/133 (88.0%) |
|
||||
| multi-session | 133 | 115/133 (86.5%) | 101/133 (75.9%) |
|
||||
| single-session-preference | 30 | 21/30 (70.0%) | 10/30 (33.3%) |
|
||||
| **Overall** | **500** | **455/500 (91.0%)** | **411/500 (82.2%)** |
|
||||
|
||||
Prompted token 消耗：总 input 13,111,421 (平均 26,275/题)，总 output 313,370 (平均 628/题)。
|
||||
平均 sessions_ingested: 44.8，dreams_triggered: 0。
|
||||
|
||||
|
|
@ -116,8 +116,7 @@ class PromptHandler:
|
|||
flags = {k: v for k, v in kwargs.items() if isinstance(v, bool)}
|
||||
formats = {k: v for k, v in kwargs.items() if not isinstance(v, bool)}
|
||||
|
||||
if flags:
|
||||
prompt = self._apply_flag_filter(prompt, flags)
|
||||
prompt = self._apply_flag_filter(prompt, flags)
|
||||
|
||||
return prompt.format(**formats).strip() if formats else prompt
|
||||
|
||||
|
|
|
|||
|
|
@ -64,7 +64,7 @@ jobs:
|
|||
index_update:
|
||||
backend: base
|
||||
description: "Manually trigger incremental index update for watched dirs."
|
||||
watch_dirs: [daily_dir, digest_dir, dialog_dir]
|
||||
watch_dirs: [daily_dir, digest_dir, session_dir/dialog]
|
||||
watch_suffixes: [md, jsonl]
|
||||
parameters:
|
||||
type: object
|
||||
|
|
|
|||
|
|
@ -62,7 +62,7 @@ jobs:
|
|||
index_update:
|
||||
backend: base
|
||||
description: "Manually trigger incremental index update for watched dirs."
|
||||
watch_dirs: [daily_dir, digest_dir, dialog_dir]
|
||||
watch_dirs: [daily_dir, digest_dir, session_dir/dialog]
|
||||
watch_suffixes: [md, jsonl]
|
||||
parameters:
|
||||
type: object
|
||||
|
|
|
|||
|
|
@ -1,8 +1,9 @@
|
|||
"""Application configuration models."""
|
||||
|
||||
import os
|
||||
from pathlib import PurePosixPath, PureWindowsPath
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
from ..enumeration import ComponentEnum
|
||||
|
||||
|
|
@ -35,11 +36,11 @@ class ApplicationConfig(BaseModel):
|
|||
workspace_dir: str = Field(default=".reme", description="Workspace root directory for runtime files")
|
||||
metadata_dir: str = Field(default="metadata", description="Subdirectory for ReMe persistent state")
|
||||
session_dir: str = Field(default="session", description="Subdirectory for persisted agent sessions")
|
||||
# dialog_dir was removed; standard transcripts are always derived as ``{session_dir}/dialog``.
|
||||
mem_session_dir: str = Field(default="mem_session", description="Subdirectory for persisted agent sessions")
|
||||
resource_dir: str = Field(default="resource", description="Subdirectory for external assets")
|
||||
daily_dir: str = Field(default="daily", description="Subdirectory for daily memory")
|
||||
digest_dir: str = Field(default="digest", description="Subdirectory for digest memory")
|
||||
dialog_dir: str = Field(default="session/dialog", description="Subdirectory for dialog session transcripts")
|
||||
enable_logo: bool = Field(default=True, description="Show ASCII logo on startup")
|
||||
timezone: str | None = Field(default="Asia/Shanghai", description="IANA timezone; None uses local time")
|
||||
language: str = Field(default="", description="Default language for LLM interactions")
|
||||
|
|
@ -53,3 +54,12 @@ class ApplicationConfig(BaseModel):
|
|||
default_factory=dict,
|
||||
description="Component registry keyed by type then name",
|
||||
)
|
||||
|
||||
@field_validator("session_dir")
|
||||
@classmethod
|
||||
def validate_session_dir(cls, value: str) -> str:
|
||||
"""Keep standard session storage inside the configured workspace."""
|
||||
path = value.strip()
|
||||
if PurePosixPath(path.replace("\\", "/")).is_absolute() or PureWindowsPath(path).anchor:
|
||||
raise ValueError("session_dir must be a workspace-relative path")
|
||||
return value
|
||||
|
|
|
|||
|
|
@ -1,25 +1,16 @@
|
|||
system_prompt: |
|
||||
You are a memory retrieval assistant. A memory system manages the user's conversation history. Given a query, you MUST call the `search` tool at least once to retrieve relevant content from the memory system before answering.
|
||||
|
||||
## Memory System
|
||||
The memory system stores the user's history in three forms:
|
||||
- **Summary notes**: condensed knowledge distilled from past sessions.
|
||||
- **Compressed session chunks**: fragments of session transcripts compressed to drop low-information phrasing; the dialogue structure is generally kept, but some detail may be shortened or lost.
|
||||
- **Original sessions**: complete verbatim transcripts of past conversations, stored as memory files.
|
||||
|
||||
The `search` tool retrieves summary notes and compressed session chunks; it does NOT return original sessions directly. Every result is a chunk — a fragment of a memory file identified by its path — and may lack surrounding context. Compressed session chunks are marked with a leading `compressed session chunk:` notice. When a chunk looks highly relevant to the query but its content is vague, truncated, or missing detail due to compression, use the `read` tool with the chunk's path to read the original session before drawing conclusions.
|
||||
|
||||
## Workflow
|
||||
1. Use `search` to locate possibly relevant content. It is a hybrid retrieval tool combining BM25 keyword matching and vector semantic search, ensuring diverse recall. Call it multiple times with different query phrasings to gather comprehensive information; do not stop after the first relevant result unless you are confident the information is sufficient.
|
||||
2. When a retrieved chunk is strongly related to the query but ambiguous because of compression, call `read` on its path to recover the full original session, and ground your answer on that.
|
||||
3. Use `add_draft` to save key findings during the search, and `read_all_draft` to review all saved notes before answering.
|
||||
4. After finding a preliminary answer, continue searching from multiple angles to check for potentially conflicting, supplementary, or updated information. The goal is to improve the accuracy and completeness of the final answer.
|
||||
5. Stop searching and generate the final answer once you are confident that further searches will not yield additional useful information.
|
||||
- Your total tool calls should be at most 9 times.
|
||||
You are a memory retrieval assistant. You MUST use the search tool to find information before answering.
|
||||
- Your total time of tool calls should be at most 9 times
|
||||
|
||||
## Search Strategy
|
||||
- You can call 'search' tool to search multiple times (at least once) with different queries to gather comprehensive information.
|
||||
- Do not stop after the first relevant one unless you are confident that the information is sufficient.
|
||||
- You can use 'read' tool to read specific parts of a specific file.
|
||||
- After finding a preliminary answer, continue searching from multiple angles to check for potentially conflicting, supplementary, or updated information.
|
||||
## Draft Tool
|
||||
- Use 'add_draft' to save key findings during search, and 'read_all_draft' to review all saved notes before answering.
|
||||
## Answer Rules
|
||||
- Answer based ONLY on retrieved or read context.
|
||||
- If memories contain contradictory information, weigh the context, query, and most importantly the timestamp of each memory to determine the final answer.
|
||||
- Answer based ONLY on retrieved context.
|
||||
- Output ONLY the direct factual answer — no reasoning, no search process, no elaboration.
|
||||
- Do NOT invent or assume information that isn't in the memories. Do NOT extrapolate or fabricate connections from related but non-direct evidence.
|
||||
- If information is not found or not sufficient after multiple searches, or if the maximum iteration count is reached without finding answer, reply: 'Information not found.'
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ from ._evolve import agent_reply_result_text, format_history, now
|
|||
from ..base_step import BaseStep
|
||||
from ..file_io import extract_daily_date, parse_daily_date, refresh_day_index
|
||||
from ..file_io import validate_filename_component, validate_session_id
|
||||
from ..index import normalize_posix_path
|
||||
from ...components import R
|
||||
|
||||
_SESSION_ID_KEY = "session_id"
|
||||
|
|
@ -66,13 +67,13 @@ class AutoMemoryStep(BaseStep):
|
|||
self.update_tools: list[str] = ["read", "edit", "frontmatter_update", "write"]
|
||||
|
||||
def _session_dir(self) -> str:
|
||||
return str(self.config_value("session_dir")).strip("/")
|
||||
return normalize_posix_path(str(self.config_value("session_dir")))
|
||||
|
||||
def _session_path(self, session_id: str) -> Path:
|
||||
return self.file_store.workspace_path / self._session_dir() / "dialog" / f"{session_id}.jsonl"
|
||||
|
||||
def _session_source_path(self, session_id: str) -> str:
|
||||
return f"{self._session_dir()}/dialog/{session_id}.jsonl"
|
||||
return normalize_posix_path(f"{self._session_dir()}/dialog/{session_id}.jsonl")
|
||||
|
||||
def _session_link(self, session_id: str) -> str:
|
||||
return f"[[{self._session_source_path(session_id)}]]"
|
||||
|
|
|
|||
|
|
@ -27,6 +27,7 @@ from pathlib import Path
|
|||
from typing import Any
|
||||
|
||||
from .auto_memory import AutoMemoryStep
|
||||
from ..index import normalize_posix_path
|
||||
from ...components import R
|
||||
from ...components.agent_wrapper import CcFileSessionStore
|
||||
|
||||
|
|
@ -73,7 +74,8 @@ class AutoMemoryCCStep(AutoMemoryStep):
|
|||
return
|
||||
|
||||
def _session_link(self, session_id: str) -> str:
|
||||
return f"[[{self._session_dir()}/{self._CC_STORE_SUBDIR}/{session_id}.jsonl]]"
|
||||
path = normalize_posix_path(f"{self._session_dir()}/{self._CC_STORE_SUBDIR}/{session_id}.jsonl")
|
||||
return f"[[{path}]]"
|
||||
|
||||
# ----- session: Claude Code side <-> ReMe CC SessionStore ----------------
|
||||
|
||||
|
|
|
|||
|
|
@ -18,8 +18,8 @@ integrate_system_prompt_procedure: |
|
|||
genuinely needs more.
|
||||
- If the draft starts copying paragraphs or narrating the event, it belongs
|
||||
in the source material, not digest.
|
||||
- The Sources section carries details: cite every relevant unit_paths entry
|
||||
as `- [[<path>]]`.
|
||||
- The Sources section carries provenance: cite every relevant unit_paths
|
||||
entry inside a complete sentence that explains what the material supports.
|
||||
- Digest-to-digest wikilinks carry the conceptual graph.
|
||||
|
||||
## Procedure Body Shape
|
||||
|
|
@ -30,9 +30,10 @@ integrate_system_prompt_procedure: |
|
|||
- Steps: numbered or terse bullets; each step is verb-led.
|
||||
- Pre-conditions / inputs: short list, not prose.
|
||||
- Failure modes / caveats: brief.
|
||||
- `## Sources`: at least one `- [[<material-path>]]` item, and normally one
|
||||
for every relevant path in unit_paths. Plain-prose provenance does NOT
|
||||
count; only wikilinks survive future updates.
|
||||
- `## Sources`: at least one sentence containing `[[<material-path>]]`, and
|
||||
normally one for every relevant path in unit_paths. A bare wikilink does
|
||||
not count; every link must have surrounding prose that explains what the
|
||||
source supports.
|
||||
|
||||
## Workflow
|
||||
|
||||
|
|
@ -60,12 +61,18 @@ integrate_system_prompt_procedure: |
|
|||
UPDATE. UPDATE must be additive: never remove existing wikilinks or
|
||||
source links. Default to weaving more, not less; this is the only
|
||||
chance to attach recalled related nodes.
|
||||
5. Before returning, read the target and rewrite any standalone relation
|
||||
field or bare-link line into a contextual sentence.
|
||||
|
||||
## Wikilink Graph
|
||||
|
||||
- A `## Sources` section points from digest back to material with bare list
|
||||
items such as `- [[daily/<date>/<session>.md]]` and
|
||||
`- [[resource/<path>]]`.
|
||||
- A `## Sources` section points from digest back to material. Each entry must
|
||||
be a complete contextual sentence, for example `- This workflow was
|
||||
observed in [[daily/<date>/<session>.md|the session record]].`
|
||||
- Never emit standalone relation fields or bare-link lines such as
|
||||
`relates_to:: [[...]]`, `derived_from:: [[...]]`, or `- [[...]]`.
|
||||
Digest links belong inside the sentence that explains the relationship;
|
||||
source links belong inside the sentence that explains the evidence.
|
||||
- Procedure nodes may link to any digest bucket:
|
||||
`[[{digest_dir}/procedure/<slug>.md]]`,
|
||||
`[[{digest_dir}/personal/<slug>.md]]`, or
|
||||
|
|
@ -98,7 +105,7 @@ integrate_system_prompt_procedure_zh: |
|
|||
|
||||
- 正文短且抽象，通常 50-200 words；只有流程本身确实需要时才更长。
|
||||
- 如果草稿开始复制段落或叙述事件，说明细节放错层了。
|
||||
- Sources 章节承载细节：用 `- [[<path>]]` 引用 unit_paths 中每个相关来源。
|
||||
- Sources 章节承载 provenance：在完整句子中引用 unit_paths 中每个相关来源，并说明材料支持什么。
|
||||
- Digest 之间的 wikilink 承载概念图。
|
||||
|
||||
## Procedure 正文形态
|
||||
|
|
@ -109,8 +116,8 @@ integrate_system_prompt_procedure_zh: |
|
|||
- Steps：编号或短 bullet；每步以动词开头。
|
||||
- Pre-conditions / inputs：短列表，不写长 prose。
|
||||
- Failure modes / caveats：简短。
|
||||
- `## Sources`：至少一条 `- [[<material-path>]]`，通常覆盖 unit_paths 中每个相关 path。
|
||||
纯文本 provenance 不算；只有 wikilink 会在未来更新中保留下来。
|
||||
- `## Sources`：至少一个包含 `[[<material-path>]]` 的完整句子，通常覆盖 unit_paths 中每个相关 path。
|
||||
裸 Wikilink 不算；每个链接周围都必须有自然语言，说明该来源支持什么。
|
||||
|
||||
## 工作流
|
||||
|
||||
|
|
@ -129,11 +136,14 @@ integrate_system_prompt_procedure_zh: |
|
|||
`> note: contradicted by [[<path>]] - <one-line>` 内联标注。
|
||||
4. CREATE 和 UPDATE 都要把 related digest 节点织入正文 wikilink。UPDATE 必须只增不删：不要删除已有
|
||||
wikilink 或来源链接。默认多织，而不是少织；这是挂接召回到的相关节点的唯一机会。
|
||||
5. 返回前读取目标文件，把任何独立关系字段或裸链接行改写成有上下文的自然句子。
|
||||
|
||||
## Wikilink 图
|
||||
|
||||
- `## Sources` 章节用普通列表 Wikilink 从 digest 指回材料，例如
|
||||
`- [[daily/<date>/<session>.md]]` 和 `- [[resource/<path>]]`。
|
||||
- `## Sources` 章节从 digest 指回材料。每一项必须是有上下文的完整句子，例如
|
||||
`- 这一流程见于 [[daily/<date>/<session>.md|该会话记录]]。`
|
||||
- 禁止生成 `relates_to:: [[...]]`、`derived_from:: [[...]]` 或 `- [[...]]` 这类独立关系字段和裸链接行。
|
||||
Digest 链接必须融入解释关系的句子；来源链接必须融入解释证据的句子。
|
||||
- Procedure 节点可以链接任意 digest bucket：
|
||||
`[[{digest_dir}/procedure/<slug>.md]]`、`[[{digest_dir}/personal/<slug>.md]]`、
|
||||
`[[{digest_dir}/wiki/<slug>.md]]`。
|
||||
|
|
@ -166,8 +176,8 @@ integrate_system_prompt_personal: |
|
|||
|
||||
- Body is short and operational, usually 50-200 words.
|
||||
- Do not narrate what the user said in detail; cite source material instead.
|
||||
- The Sources section carries details: cite every relevant unit_paths entry
|
||||
as `- [[<path>]]`.
|
||||
- The Sources section carries provenance: cite every relevant unit_paths
|
||||
entry inside a complete sentence that explains what the material supports.
|
||||
- Digest-to-digest wikilinks carry the conceptual graph.
|
||||
|
||||
## Personal Body Shape
|
||||
|
|
@ -180,9 +190,10 @@ integrate_system_prompt_personal: |
|
|||
- `How to apply:` contexts, tasks, boundaries, or exceptions.
|
||||
- Do not invent exceptions or soften a hard preference unless the source
|
||||
material explicitly supports that exception.
|
||||
- `## Sources`: at least one `- [[<material-path>]]` item, and normally one
|
||||
for every relevant path in unit_paths. Plain-prose provenance does NOT
|
||||
count; only wikilinks survive future updates.
|
||||
- `## Sources`: at least one sentence containing `[[<material-path>]]`, and
|
||||
normally one for every relevant path in unit_paths. A bare wikilink does
|
||||
not count; every link must have surrounding prose that explains what the
|
||||
source supports.
|
||||
|
||||
For preferences, prefer one node per preference rather than one large person
|
||||
node; that is the granularity downstream search will hit.
|
||||
|
|
@ -213,12 +224,18 @@ integrate_system_prompt_personal: |
|
|||
UPDATE. UPDATE must be additive: never remove existing wikilinks or
|
||||
source links. Default to weaving more, not less; this is the only
|
||||
chance to attach recalled related nodes.
|
||||
5. Before returning, read the target and rewrite any standalone relation
|
||||
field or bare-link line into a contextual sentence.
|
||||
|
||||
## Wikilink Graph
|
||||
|
||||
- A `## Sources` section points from digest back to material with bare list
|
||||
items such as `- [[daily/<date>/<session>.md]]` and
|
||||
`- [[resource/<path>]]`.
|
||||
- A `## Sources` section points from digest back to material. Each entry must
|
||||
be a complete contextual sentence, for example `- This preference was
|
||||
observed in [[daily/<date>/<session>.md|the session record]].`
|
||||
- Never emit standalone relation fields or bare-link lines such as
|
||||
`relates_to:: [[...]]`, `derived_from:: [[...]]`, or `- [[...]]`.
|
||||
Digest links belong inside the sentence that explains the relationship;
|
||||
source links belong inside the sentence that explains the evidence.
|
||||
- Personal nodes may link to any digest bucket:
|
||||
`[[{digest_dir}/personal/<slug>.md]]`,
|
||||
`[[{digest_dir}/procedure/<slug>.md]]`, or
|
||||
|
|
@ -252,7 +269,7 @@ integrate_system_prompt_personal_zh: |
|
|||
|
||||
- 正文短且可操作，通常 50-200 words。
|
||||
- 不要详细复述用户说了什么；用来源材料承载细节。
|
||||
- Sources 章节承载细节：用 `- [[<path>]]` 引用 unit_paths 中每个相关来源。
|
||||
- Sources 章节承载 provenance：在完整句子中引用 unit_paths 中每个相关来源，并说明材料支持什么。
|
||||
- Digest 之间的 wikilink 承载概念图。
|
||||
|
||||
## Personal 正文形态
|
||||
|
|
@ -263,8 +280,8 @@ integrate_system_prompt_personal_zh: |
|
|||
- `Why:` 原因或上下文，帮助未来判断边界情况。
|
||||
- `How to apply:` 适用上下文、任务、边界或例外。
|
||||
- 不要凭空添加例外，也不要软化明确偏好；只有来源材料明确支持时才写例外。
|
||||
- `## Sources`：至少一条 `- [[<material-path>]]`，通常覆盖 unit_paths 中每个相关 path。
|
||||
纯文本 provenance 不算；只有 wikilink 会在未来更新中保留下来。
|
||||
- `## Sources`：至少一个包含 `[[<material-path>]]` 的完整句子，通常覆盖 unit_paths 中每个相关 path。
|
||||
裸 Wikilink 不算；每个链接周围都必须有自然语言，说明该来源支持什么。
|
||||
|
||||
偏好类内容优先一条偏好一个 node，而不是一个人一个大 node；这是下游搜索更容易命中的粒度。
|
||||
|
||||
|
|
@ -285,11 +302,14 @@ integrate_system_prompt_personal_zh: |
|
|||
`> note: contradicted by [[<path>]] - <one-line>` 内联标注。
|
||||
4. CREATE 和 UPDATE 都要把 related digest 节点织入正文 wikilink。UPDATE 必须只增不删：不要删除已有
|
||||
wikilink 或来源链接。默认多织，而不是少织；这是挂接召回到的相关节点的唯一机会。
|
||||
5. 返回前读取目标文件，把任何独立关系字段或裸链接行改写成有上下文的自然句子。
|
||||
|
||||
## Wikilink 图
|
||||
|
||||
- `## Sources` 章节用普通列表 Wikilink 从 digest 指回材料，例如
|
||||
`- [[daily/<date>/<session>.md]]` 和 `- [[resource/<path>]]`。
|
||||
- `## Sources` 章节从 digest 指回材料。每一项必须是有上下文的完整句子，例如
|
||||
`- 这一偏好见于 [[daily/<date>/<session>.md|该会话记录]]。`
|
||||
- 禁止生成 `relates_to:: [[...]]`、`derived_from:: [[...]]` 或 `- [[...]]` 这类独立关系字段和裸链接行。
|
||||
Digest 链接必须融入解释关系的句子；来源链接必须融入解释证据的句子。
|
||||
- Personal 节点可以链接任意 digest bucket：
|
||||
`[[{digest_dir}/personal/<slug>.md]]`、`[[{digest_dir}/procedure/<slug>.md]]`、
|
||||
`[[{digest_dir}/wiki/<slug>.md]]`。
|
||||
|
|
@ -325,8 +345,8 @@ integrate_system_prompt_wiki: |
|
|||
genuinely needs more.
|
||||
- If the draft starts copying paragraphs or narrating the event, it belongs
|
||||
in the source material, not digest.
|
||||
- The Sources section carries details: cite every relevant unit_paths entry
|
||||
as `- [[<path>]]`.
|
||||
- The Sources section carries provenance: cite every relevant unit_paths
|
||||
entry inside a complete sentence that explains what the material supports.
|
||||
- Digest-to-digest wikilinks carry the conceptual graph.
|
||||
|
||||
## Wiki Body Shape
|
||||
|
|
@ -337,9 +357,10 @@ integrate_system_prompt_wiki: |
|
|||
- Body: short paragraphs or tight bullets with properties, sub-claims,
|
||||
distinctions, and one-line examples.
|
||||
- Relations: weave related nodes into natural prose with ordinary wikilinks.
|
||||
- `## Sources`: at least one `- [[<material-path>]]` item, and normally one
|
||||
for every relevant path in unit_paths. Plain-prose provenance does NOT
|
||||
count; only wikilinks survive future updates.
|
||||
- `## Sources`: at least one sentence containing `[[<material-path>]]`, and
|
||||
normally one for every relevant path in unit_paths. A bare wikilink does
|
||||
not count; every link must have surrounding prose that explains what the
|
||||
source supports.
|
||||
|
||||
## Workflow
|
||||
|
||||
|
|
@ -368,12 +389,18 @@ integrate_system_prompt_wiki: |
|
|||
UPDATE. UPDATE must be additive: never remove existing wikilinks or
|
||||
source links. Default to weaving more, not less; this is the only
|
||||
chance to attach recalled related nodes.
|
||||
5. Before returning, read the target and rewrite any standalone relation
|
||||
field or bare-link line into a contextual sentence.
|
||||
|
||||
## Wikilink Graph
|
||||
|
||||
- A `## Sources` section points from digest back to material with bare list
|
||||
items such as `- [[daily/<date>/<session>.md]]` and
|
||||
`- [[resource/<path>]]`.
|
||||
- A `## Sources` section points from digest back to material. Each entry must
|
||||
be a complete contextual sentence, for example `- This claim is supported
|
||||
by [[daily/<date>/<session>.md|the session record]].`
|
||||
- Never emit standalone relation fields or bare-link lines such as
|
||||
`relates_to:: [[...]]`, `derived_from:: [[...]]`, or `- [[...]]`.
|
||||
Digest links belong inside the sentence that explains the relationship;
|
||||
source links belong inside the sentence that explains the evidence.
|
||||
- Wiki nodes may link to any digest bucket:
|
||||
`[[{digest_dir}/wiki/<slug>.md]]`,
|
||||
`[[{digest_dir}/procedure/<slug>.md]]`, or
|
||||
|
|
@ -407,7 +434,7 @@ integrate_system_prompt_wiki_zh: |
|
|||
|
||||
- 正文短且抽象，通常 50-200 words；只有概念本身确实需要时才更长。
|
||||
- 如果草稿开始复制段落或叙述事件，说明细节放错层了。
|
||||
- Sources 章节承载细节：用 `- [[<path>]]` 引用 unit_paths 中每个相关来源。
|
||||
- Sources 章节承载 provenance：在完整句子中引用 unit_paths 中每个相关来源，并说明材料支持什么。
|
||||
- Digest 之间的 wikilink 承载概念图。
|
||||
|
||||
## Wiki 正文形态
|
||||
|
|
@ -417,8 +444,8 @@ integrate_system_prompt_wiki_zh: |
|
|||
- First line：一句话定义或主张。
|
||||
- Body：短段落或紧凑 bullets，写属性、子主张、区分和一行例子。
|
||||
- Relations：在自然语言中用普通 Wikilink 织入相关节点。
|
||||
- `## Sources`：至少一条 `- [[<material-path>]]`，通常覆盖 unit_paths 中每个相关 path。
|
||||
纯文本 provenance 不算；只有 wikilink 会在未来更新中保留下来。
|
||||
- `## Sources`：至少一个包含 `[[<material-path>]]` 的完整句子，通常覆盖 unit_paths 中每个相关 path。
|
||||
裸 Wikilink 不算；每个链接周围都必须有自然语言，说明该来源支持什么。
|
||||
|
||||
## 工作流
|
||||
|
||||
|
|
@ -437,11 +464,14 @@ integrate_system_prompt_wiki_zh: |
|
|||
`> note: contradicted by [[<path>]] - <one-line>` 内联标注。
|
||||
4. CREATE 和 UPDATE 都要把 related digest 节点织入正文 wikilink。UPDATE 必须只增不删：不要删除已有
|
||||
wikilink 或来源链接。默认多织，而不是少织；这是挂接召回到的相关节点的唯一机会。
|
||||
5. 返回前读取目标文件，把任何独立关系字段或裸链接行改写成有上下文的自然句子。
|
||||
|
||||
## Wikilink 图
|
||||
|
||||
- `## Sources` 章节用普通列表 Wikilink 从 digest 指回材料，例如
|
||||
`- [[daily/<date>/<session>.md]]` 和 `- [[resource/<path>]]`。
|
||||
- `## Sources` 章节从 digest 指回材料。每一项必须是有上下文的完整句子，例如
|
||||
`- 这一主张由 [[daily/<date>/<session>.md|该会话记录]] 支持。`
|
||||
- 禁止生成 `relates_to:: [[...]]`、`derived_from:: [[...]]` 或 `- [[...]]` 这类独立关系字段和裸链接行。
|
||||
Digest 链接必须融入解释关系的句子；来源链接必须融入解释证据的句子。
|
||||
- Wiki 节点可以链接任意 digest bucket：
|
||||
`[[{digest_dir}/wiki/<slug>.md]]`、`[[{digest_dir}/procedure/<slug>.md]]`、
|
||||
`[[{digest_dir}/personal/<slug>.md]]`。
|
||||
|
|
@ -471,8 +501,9 @@ integrate_user_message: |
|
|||
{material_blob}
|
||||
|
||||
Integrate this single unit into digest memory. Cite every relevant unit_paths
|
||||
entry as `- [[<path>]]` under `## Sources`, recall related digest nodes, and
|
||||
weave useful digest wikilinks into the target node.
|
||||
entry in a contextual sentence under `## Sources`, recall related digest
|
||||
nodes, and weave useful digest wikilinks into natural body prose. Never emit
|
||||
standalone relation fields or bare-link lines.
|
||||
|
||||
integrate_user_message_zh: |
|
||||
提示：{hint}
|
||||
|
|
@ -488,5 +519,6 @@ integrate_user_message_zh: |
|
|||
|
||||
{material_blob}
|
||||
|
||||
将这个单独的 unit 整合进 digest memory。在 `## Sources` 下用 `- [[<path>]]` 引用 unit_paths 中每个
|
||||
相关来源，召回相关 digest 节点，并把有用的 digest wikilink 织入目标节点。
|
||||
将这个单独的 unit 整合进 digest memory。在 `## Sources` 下用有上下文的完整句子引用 unit_paths 中每个
|
||||
相关来源，召回相关 digest 节点，并把有用的 digest Wikilink 融入正文的自然表达。禁止生成独立关系字段
|
||||
或裸链接行。
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@
|
|||
from ._daily_index import parse_daily_date, refresh_day_index, validate_session_id
|
||||
from ._path import validate_filename_component
|
||||
from ..base_step import BaseStep
|
||||
from ..index import normalize_posix_path
|
||||
from ...components import R
|
||||
from ...steps.evolve import now
|
||||
|
||||
|
|
@ -21,10 +22,11 @@ class DailyWriteStep(BaseStep):
|
|||
self.context.response.metadata.update(meta)
|
||||
|
||||
def _session_dir(self) -> str:
|
||||
return str(self.config_value("session_dir")).strip("/")
|
||||
return normalize_posix_path(str(self.config_value("session_dir")))
|
||||
|
||||
def _session_link(self, session_id: str) -> str:
|
||||
return f"[[{self._session_dir()}/dialog/{session_id}.jsonl]]"
|
||||
path = normalize_posix_path(f"{self._session_dir()}/dialog/{session_id}.jsonl")
|
||||
return f"[[{path}]]"
|
||||
|
||||
def _collect_required(self) -> tuple[str, str, str, str] | None:
|
||||
assert self.context is not None
|
||||
|
|
|
|||
|
|
@ -109,13 +109,13 @@ class ReadStep(BaseStep):
|
|||
the workspace or is not a raw session transcript.
|
||||
"""
|
||||
try:
|
||||
rel_path = str(target.relative_to(self.workspace_path))
|
||||
rel_path = target.relative_to(self.workspace_path).as_posix()
|
||||
except ValueError:
|
||||
self.logger.info(f"[{self.name}] skip session format: path outside workspace_path path={target}")
|
||||
return truncate_text_output(excerpt, start_line=start_line, total_lines=total, file_path=str(target))
|
||||
|
||||
dialog_dir = self.config_value("dialog_dir")
|
||||
if not is_session_path(rel_path, dialog_dir):
|
||||
session_dir = self.config_value("session_dir")
|
||||
if not is_session_path(rel_path, session_dir):
|
||||
return truncate_text_output(excerpt, start_line=start_line, total_lines=total, file_path=str(target))
|
||||
|
||||
return truncate_session_output(
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
"""Index steps."""
|
||||
|
||||
from ._source_format import normalize_posix_path
|
||||
from .bm25_search import Bm25SearchStep
|
||||
from .clear_paths import ClearPathsStep
|
||||
from .clear_store import ClearStoreStep
|
||||
|
|
@ -35,6 +36,7 @@ __all__ = [
|
|||
"InitChangesStep",
|
||||
"LogChangesStep",
|
||||
"NodeSearchStep",
|
||||
"normalize_posix_path",
|
||||
"ReadAllDraftStep",
|
||||
"OptimizeIndexStep",
|
||||
"SearchStep",
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@ removes every previously-returned result. :data:`NO_RESULTS_MESSAGE` is the
|
|||
English notice shown when the search returned no results at all.
|
||||
"""
|
||||
|
||||
import posixpath
|
||||
from typing import Callable, Final
|
||||
|
||||
from agentscope.message import Msg
|
||||
|
|
@ -38,21 +39,28 @@ ALL_RETURNED_MESSAGE: Final[str] = (
|
|||
NO_RESULTS_MESSAGE: Final[str] = "No relevant information was found for the given query."
|
||||
|
||||
|
||||
def is_session_path(path: str, dialog_dir: str) -> bool:
|
||||
"""True if the path points at a raw session transcript (a jsonl file under the dialog dir)."""
|
||||
path = (path or "").strip().strip("/")
|
||||
def normalize_posix_path(path: str) -> str:
|
||||
"""Return a normalized, workspace-relative POSIX path string."""
|
||||
normalized = posixpath.normpath((path or "").strip().replace("\\", "/").strip("/"))
|
||||
return "" if normalized == "." else normalized
|
||||
|
||||
|
||||
def is_session_path(path: str, session_dir: str) -> bool:
|
||||
"""True if the path points at a raw transcript under ``{session_dir}/dialog``."""
|
||||
path = normalize_posix_path(path)
|
||||
if not path.endswith(".jsonl"):
|
||||
return False
|
||||
dialog_dir = (dialog_dir or "").strip("/")
|
||||
return path == dialog_dir or path.startswith(f"{dialog_dir}/")
|
||||
session_root = normalize_posix_path(session_dir)
|
||||
session_dialog_dir = posixpath.join(session_root, "dialog")
|
||||
return path == session_dialog_dir or path.startswith(f"{session_dialog_dir}/")
|
||||
|
||||
|
||||
def is_session_chunk(chunk: FileChunk, dialog_dir: str) -> bool:
|
||||
def is_session_chunk(chunk: FileChunk, session_dir: str) -> bool:
|
||||
"""True if the chunk comes from a raw session transcript (a jsonl file under the dialog dir)."""
|
||||
return is_session_path(chunk.path, dialog_dir)
|
||||
return is_session_path(chunk.path, session_dir)
|
||||
|
||||
|
||||
def render_chunk_body(chunk: FileChunk, dialog_dir: str) -> str:
|
||||
def render_chunk_body(chunk: FileChunk, session_dir: str) -> str:
|
||||
"""Render a chunk's body; raw session transcripts render one message per line.
|
||||
|
||||
Session chunks are jsonl where each line is a serialized ``Msg``. They
|
||||
|
|
@ -61,7 +69,7 @@ def render_chunk_body(chunk: FileChunk, dialog_dir: str) -> str:
|
|||
share the same single-line-per-message format. All other chunks keep
|
||||
their raw ``text``.
|
||||
"""
|
||||
if not is_session_chunk(chunk, dialog_dir):
|
||||
if not is_session_chunk(chunk, session_dir):
|
||||
return chunk.text
|
||||
return "\n".join(render_session_chunk_lines(chunk))
|
||||
|
||||
|
|
@ -123,7 +131,7 @@ def _build_union_chunk(group: list[FileChunk]) -> FileChunk:
|
|||
)
|
||||
|
||||
|
||||
def merge_session_chunk_intervals(chunks: list[FileChunk], dialog_dir: str) -> list[FileChunk]:
|
||||
def merge_session_chunk_intervals(chunks: list[FileChunk], session_dir: str) -> list[FileChunk]:
|
||||
"""Merge raw session chunks from the same file into their line-range union.
|
||||
|
||||
Only chunks recognized as raw session transcripts (see
|
||||
|
|
@ -148,7 +156,7 @@ def merge_session_chunk_intervals(chunks: list[FileChunk], dialog_dir: str) -> l
|
|||
# non-session chunks use their own rank and thus keep their position.
|
||||
ordered: list[tuple[int, int, FileChunk]] = []
|
||||
for idx, c in enumerate(chunks):
|
||||
if is_session_chunk(c, dialog_dir):
|
||||
if is_session_chunk(c, session_dir):
|
||||
session_by_path.setdefault(c.path, []).append((idx, c))
|
||||
else:
|
||||
ordered.append((idx, c.start_line, c))
|
||||
|
|
@ -183,7 +191,7 @@ def _finalize_group(group: list[FileChunk]) -> FileChunk:
|
|||
|
||||
def render_chunk_entries(
|
||||
chunks: list[FileChunk],
|
||||
dialog_dir: str,
|
||||
session_dir: str,
|
||||
*,
|
||||
include_source: bool = True,
|
||||
score_fn: Callable[[FileChunk], str] | None = None,
|
||||
|
|
@ -207,7 +215,7 @@ def render_chunk_entries(
|
|||
expansion lines (used by hybrid search).
|
||||
"""
|
||||
if not include_source:
|
||||
return [{"body": render_chunk_body(c, dialog_dir)} for c in chunks]
|
||||
return [{"body": render_chunk_body(c, session_dir)} for c in chunks]
|
||||
|
||||
fmt = score_fn or (lambda c: f"score={c.score:.4f}")
|
||||
entries: list[dict[str, str]] = []
|
||||
|
|
@ -218,7 +226,7 @@ def render_chunk_entries(
|
|||
"start_line": str(c.start_line),
|
||||
"end_line": str(c.end_line),
|
||||
"score": fmt(c),
|
||||
"body": render_chunk_body(c, dialog_dir),
|
||||
"body": render_chunk_body(c, session_dir),
|
||||
"link": "\n".join(render_expansion_lines((link_expansion or {}).get(c.path, {}))),
|
||||
},
|
||||
)
|
||||
|
|
|
|||
|
|
@ -4,6 +4,8 @@ from dataclasses import dataclass, field
|
|||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from ._source_format import normalize_posix_path
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from ...schema import ApplicationConfig
|
||||
from ...components.runtime_context import RuntimeContext
|
||||
|
|
@ -27,8 +29,19 @@ def build_watch_rules(
|
|||
"""Build watch rules from application config fields and suffix whitelist."""
|
||||
rules: list[WatchRule] = []
|
||||
for dir_field in watch_dirs:
|
||||
dir_name = getattr(app_config, dir_field, dir_field)
|
||||
rules.append(WatchRule(path=workspace_path / dir_name, suffixes=list(watch_suffixes)))
|
||||
literal_path = Path(dir_field)
|
||||
if literal_path.is_absolute():
|
||||
rule_path = literal_path
|
||||
else:
|
||||
config_field, separator, child_path = dir_field.partition("/")
|
||||
dir_value = getattr(app_config, config_field, config_field)
|
||||
if hasattr(app_config, config_field) and dir_value in (None, ""):
|
||||
dir_value = getattr(type(app_config)(), config_field)
|
||||
if separator:
|
||||
dir_value = normalize_posix_path(f"{dir_value}/{child_path}")
|
||||
dir_name = Path(dir_value)
|
||||
rule_path = dir_name if dir_name.is_absolute() else workspace_path / dir_name
|
||||
rules.append(WatchRule(path=rule_path, suffixes=list(watch_suffixes)))
|
||||
return rules
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -48,10 +48,10 @@ class Bm25SearchStep(_ToolContextDedupMixin, BaseStep):
|
|||
else:
|
||||
results = results[:limit]
|
||||
|
||||
dialog_dir = self.config_value("dialog_dir")
|
||||
session_dir = self.config_value("session_dir")
|
||||
entries = render_chunk_entries(
|
||||
merge_session_chunk_intervals(results, dialog_dir),
|
||||
dialog_dir,
|
||||
merge_session_chunk_intervals(results, session_dir),
|
||||
session_dir,
|
||||
include_source=self.include_source,
|
||||
)
|
||||
self.context.response.answer = join_chunk_entries(entries)
|
||||
|
|
|
|||
|
|
@ -213,15 +213,15 @@ class SearchV2Step(_ToolContextDedupMixin, BaseStep):
|
|||
await expand_links(self.file_store, unique_paths, max_links_per_direction) if expand_links_enabled else {}
|
||||
)
|
||||
|
||||
dialog_dir = self.config_value("dialog_dir")
|
||||
session_dir = self.config_value("session_dir")
|
||||
entries = render_chunk_entries(
|
||||
merge_session_chunk_intervals(fused, dialog_dir),
|
||||
dialog_dir,
|
||||
merge_session_chunk_intervals(fused, session_dir),
|
||||
session_dir,
|
||||
score_fn=lambda c: self._format_scores(c.scores, hybrid),
|
||||
link_expansion=link_expansion,
|
||||
)
|
||||
if self._session_compress_enabled():
|
||||
await self._compress_session_entries(entries, query, dialog_dir)
|
||||
await self._compress_session_entries(entries, query, session_dir)
|
||||
self.context.response.answer = join_chunk_entries(entries)
|
||||
if not fused:
|
||||
self.context.response.answer = ALL_RETURNED_MESSAGE if pre_dedup_count > 0 else NO_RESULTS_MESSAGE
|
||||
|
|
@ -246,7 +246,7 @@ class SearchV2Step(_ToolContextDedupMixin, BaseStep):
|
|||
value = (search_cfg.get("_compress") or {}).get("session")
|
||||
return value is True or str(value).strip().lower() == "true"
|
||||
|
||||
async def _compress_session_entries(self, entries: list[dict[str, str]], query: str, dialog_dir: str) -> None:
|
||||
async def _compress_session_entries(self, entries: list[dict[str, str]], query: str, session_dir: str) -> None:
|
||||
"""Compress session-transcript entry bodies in place via the ``compressor`` job.
|
||||
|
||||
Only entries whose ``path`` points at a raw session transcript are
|
||||
|
|
@ -323,7 +323,7 @@ class SearchV2Step(_ToolContextDedupMixin, BaseStep):
|
|||
return
|
||||
entry["body"] = f"compressed session chunk:\n{compressed}"
|
||||
|
||||
targets = [e for e in entries if is_session_path(e.get("path", ""), dialog_dir)]
|
||||
targets = [e for e in entries if is_session_path(e.get("path", ""), session_dir)]
|
||||
if not targets:
|
||||
return
|
||||
self.logger.info(f"[{self.name}] compressing {len(targets)} session entries with {len(queries)} queries")
|
||||
|
|
|
|||
|
|
@ -48,10 +48,10 @@ class VectorSearchStep(_ToolContextDedupMixin, BaseStep):
|
|||
else:
|
||||
results = results[:limit]
|
||||
|
||||
dialog_dir = self.config_value("dialog_dir")
|
||||
session_dir = self.config_value("session_dir")
|
||||
entries = render_chunk_entries(
|
||||
merge_session_chunk_intervals(results, dialog_dir),
|
||||
dialog_dir,
|
||||
merge_session_chunk_intervals(results, session_dir),
|
||||
session_dir,
|
||||
include_source=self.include_source,
|
||||
)
|
||||
self.context.response.answer = join_chunk_entries(entries)
|
||||
|
|
|
|||
148
tests/unit/test_application_config.py
Normal file
148
tests/unit/test_application_config.py
Normal file
|
|
@ -0,0 +1,148 @@
|
|||
"""Application path configuration contracts."""
|
||||
|
||||
# pylint: disable=protected-access
|
||||
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
from agentscope.message import Msg
|
||||
from pydantic import ValidationError
|
||||
|
||||
from reme.application import Application
|
||||
from reme.components import ApplicationContext
|
||||
from reme.schema import ApplicationConfig, FileChunk
|
||||
from reme.steps.evolve.auto_memory import AutoMemoryStep
|
||||
from reme.steps.file_io.daily_write import DailyWriteStep
|
||||
from reme.steps.index._source_format import is_session_path, render_chunk_body
|
||||
from reme.steps.index._watch_rules import build_watch_rules
|
||||
|
||||
|
||||
def test_dialog_dir_is_not_an_application_config_field():
|
||||
"""The removed option is absent from schemas and ignored when supplied."""
|
||||
custom = ApplicationConfig(session_dir="sessions/", dialog_dir="somewhere/else")
|
||||
|
||||
assert not hasattr(custom, "dialog_dir")
|
||||
assert "dialog_dir" not in custom.model_dump()
|
||||
assert "dialog_dir" not in ApplicationConfig.model_json_schema()["properties"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("session_dir", ["/tmp/sessions", "C:\\sessions", "\\\\server\\share\\sessions"])
|
||||
def test_application_rejects_absolute_session_dir_before_workspace_setup(tmp_path, session_dir):
|
||||
"""Application initialization rejects session directories outside the workspace."""
|
||||
workspace = tmp_path / "workspace"
|
||||
|
||||
with pytest.raises(ValidationError, match="session_dir must be a workspace-relative path"):
|
||||
Application(workspace_dir=str(workspace), session_dir=session_dir)
|
||||
|
||||
assert not workspace.exists()
|
||||
|
||||
|
||||
def test_dialog_watch_rule_follows_custom_session_dir(tmp_path):
|
||||
"""Nested watch rules resolve the dialog directory below session_dir."""
|
||||
config = ApplicationConfig(session_dir="sessions")
|
||||
|
||||
rules = build_watch_rules(
|
||||
config,
|
||||
tmp_path,
|
||||
watch_dirs=["session_dir/dialog"],
|
||||
watch_suffixes=["jsonl"],
|
||||
)
|
||||
|
||||
assert len(rules) == 1
|
||||
assert rules[0].path == tmp_path / "sessions" / "dialog"
|
||||
assert rules[0].suffixes == ["jsonl"]
|
||||
|
||||
|
||||
def test_dialog_watch_rule_matches_writer_for_normalized_session_dirs(tmp_path):
|
||||
"""Nested watch rules use the same defaults and normalization as transcript writers."""
|
||||
for configured in ("", "parent//../sessions"):
|
||||
app_context = ApplicationContext(session_dir=configured)
|
||||
auto_memory = AutoMemoryStep(app_context=app_context)
|
||||
auto_memory.file_store = SimpleNamespace(workspace_path=tmp_path)
|
||||
|
||||
rules = build_watch_rules(
|
||||
app_context.app_config,
|
||||
tmp_path,
|
||||
watch_dirs=["session_dir/dialog"],
|
||||
watch_suffixes=["jsonl"],
|
||||
)
|
||||
|
||||
assert rules[0].path == auto_memory._session_path("s1").parent
|
||||
|
||||
|
||||
def test_absolute_watch_rule_stays_outside_workspace(tmp_path):
|
||||
"""Absolute literal watch directories retain their original location."""
|
||||
config = ApplicationConfig()
|
||||
absolute_dir = tmp_path.parent / "reme-dialogs"
|
||||
|
||||
rules = build_watch_rules(
|
||||
config,
|
||||
tmp_path,
|
||||
watch_dirs=[str(absolute_dir)],
|
||||
watch_suffixes=["jsonl"],
|
||||
)
|
||||
|
||||
assert rules[0].path == absolute_dir
|
||||
|
||||
|
||||
def test_standard_session_paths_and_links_follow_custom_session_dir(tmp_path):
|
||||
"""Writers derive transcript paths and links from session_dir."""
|
||||
app_context = ApplicationContext(session_dir="sessions")
|
||||
auto_memory = AutoMemoryStep(app_context=app_context)
|
||||
auto_memory.file_store = SimpleNamespace(workspace_path=tmp_path)
|
||||
daily_write = DailyWriteStep(app_context=app_context)
|
||||
|
||||
assert auto_memory._session_path("s1") == tmp_path / "sessions" / "dialog" / "s1.jsonl"
|
||||
assert auto_memory._session_source_path("s1") == "sessions/dialog/s1.jsonl"
|
||||
assert auto_memory._session_link("s1") == "[[sessions/dialog/s1.jsonl]]"
|
||||
assert daily_write._session_link("s1") == "[[sessions/dialog/s1.jsonl]]"
|
||||
|
||||
|
||||
def test_session_paths_and_links_normalize_custom_session_dir(tmp_path):
|
||||
"""Writers and source links use the same normalized POSIX session root."""
|
||||
for configured, expected in (
|
||||
("./sessions", "sessions"),
|
||||
(".", ""),
|
||||
("parent//../sessions", "sessions"),
|
||||
):
|
||||
app_context = ApplicationContext(session_dir=configured)
|
||||
auto_memory = AutoMemoryStep(app_context=app_context)
|
||||
auto_memory.file_store = SimpleNamespace(workspace_path=tmp_path)
|
||||
daily_write = DailyWriteStep(app_context=app_context)
|
||||
source_path = f"{expected + '/' if expected else ''}dialog/s1.jsonl"
|
||||
|
||||
assert auto_memory._session_path("s1") == tmp_path / source_path
|
||||
assert auto_memory._session_source_path("s1") == source_path
|
||||
assert auto_memory._session_link("s1") == f"[[{source_path}]]"
|
||||
assert daily_write._session_link("s1") == f"[[{source_path}]]"
|
||||
|
||||
|
||||
def test_dialog_classification_excludes_other_session_stores():
|
||||
"""Only jsonl files below the dialog child directory are transcripts."""
|
||||
assert is_session_path("sessions/dialog/s1.jsonl", "sessions")
|
||||
assert not is_session_path("sessions/claude_code/s1.jsonl", "sessions")
|
||||
assert not is_session_path("session/dialog/s1.jsonl", "sessions")
|
||||
|
||||
|
||||
def test_dialog_classification_normalizes_paths():
|
||||
"""Session recognition compares normalized path representations."""
|
||||
assert is_session_path("sessions/dialog/s1.jsonl", "./sessions")
|
||||
assert is_session_path("./sessions//dialog/s1.jsonl", "sessions")
|
||||
assert is_session_path("dialog/s1.jsonl", ".")
|
||||
assert is_session_path("sessions/dialog/s1.jsonl", "parent/../sessions")
|
||||
|
||||
|
||||
def test_session_rendering_follows_custom_session_dir():
|
||||
"""Session-aware rendering recognizes a custom session root."""
|
||||
message = Msg(name="user", role="user", content=[{"type": "text", "text": "hello"}])
|
||||
chunk = FileChunk(
|
||||
path="sessions/dialog/s1.jsonl",
|
||||
start_line=1,
|
||||
end_line=1,
|
||||
text=message.model_dump_json(),
|
||||
)
|
||||
|
||||
rendered = render_chunk_body(chunk, "sessions")
|
||||
|
||||
assert rendered.startswith("[user @")
|
||||
assert rendered.endswith("] hello")
|
||||
|
|
@ -156,6 +156,22 @@ def test_parse_structured_reply_handles_fenced_yaml_and_scalar_fallback():
|
|||
assert data["note"].startswith("Extended node")
|
||||
|
||||
|
||||
def test_integrate_prompts_require_wikilinks_in_contextual_prose():
|
||||
"""Every digest bucket rejects standalone relation fields and bare links."""
|
||||
prompt_path = Path(__file__).parents[2] / "reme" / "steps" / "evolve" / "dream" / "integrate.yaml"
|
||||
prompts = yaml.safe_load(prompt_path.read_text(encoding="utf-8"))
|
||||
|
||||
for bucket in ("procedure", "personal", "wiki"):
|
||||
english = prompts[f"integrate_system_prompt_{bucket}"]
|
||||
chinese = prompts[f"integrate_system_prompt_{bucket}_zh"]
|
||||
assert "Never emit standalone relation fields or bare-link lines" in english
|
||||
assert "Digest links belong inside the sentence" in english
|
||||
assert "Before returning, read the target" in english
|
||||
assert "禁止生成 `relates_to:: [[...]]`" in chinese
|
||||
assert "Digest 链接必须融入解释关系的句子" in chinese
|
||||
assert "返回前读取目标文件" in chinese
|
||||
|
||||
|
||||
def test_extract_clean_output_respects_max_units():
|
||||
"""Extract cleaning caps valid units at max_units."""
|
||||
state = DreamState(changed_paths=["daily/a.md"])
|
||||
|
|
|
|||
|
|
@ -31,7 +31,9 @@ import os
|
|||
import tempfile
|
||||
import warnings
|
||||
from pathlib import Path, PureWindowsPath
|
||||
from types import SimpleNamespace
|
||||
|
||||
from reme.components import ApplicationContext
|
||||
from reme.components.file_store import LocalFileStore
|
||||
from reme.schema import FileNode
|
||||
from reme.steps.file_io import (
|
||||
|
|
@ -811,6 +813,31 @@ def test_read_format_session_injected_false_disables_yaml_true():
|
|||
_run(run())
|
||||
|
||||
|
||||
def test_read_format_session_with_non_normalized_session_dir():
|
||||
"""Session formatting recognizes a normalized target when config contains ``./``."""
|
||||
|
||||
async def run():
|
||||
from agentscope.message import Msg
|
||||
|
||||
m = Msg(name="user", role="user", content=[{"type": "text", "text": "hello world"}])
|
||||
jsonl_body = m.model_dump_json() + "\n"
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
workspace = Path(tmp)
|
||||
session_path = workspace / "sessions" / "dialog" / "s.jsonl"
|
||||
session_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
session_path.write_text(jsonl_body, encoding="utf-8")
|
||||
|
||||
app_context = ApplicationContext(workspace_dir=str(workspace.resolve()), session_dir="./sessions")
|
||||
step = crud_read.ReadStep(app_context=app_context, read_step_format_session=True)
|
||||
step.file_store = SimpleNamespace(workspace_path=workspace)
|
||||
await step(path="sessions/dialog/s.jsonl")
|
||||
|
||||
assert "[user @" in str(step.context.response.answer)
|
||||
|
||||
_run(run())
|
||||
|
||||
|
||||
# -- write / edit --------------------------------------------------------
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -182,13 +182,18 @@ def test_flag_filter_removes_non_matching():
|
|||
def test_flag_filter_default_false():
|
||||
ph = PromptHandler()
|
||||
ph.load_prompt_dict({"p": "[debug] debug info\nbase"})
|
||||
# When no flags are passed at all, _apply_flag_filter is not called,
|
||||
# so flagged lines are kept as-is (including the tag text after regex sub).
|
||||
result = ph.prompt_format("p", debug=False)
|
||||
result = ph.prompt_format("p")
|
||||
assert "debug info" not in result
|
||||
assert "base" in result
|
||||
|
||||
|
||||
def test_flag_filter_defaults_false_with_format_variables():
|
||||
ph = PromptHandler()
|
||||
ph.load_prompt_dict({"p": "[debug] debug {name}\nhello {name}"})
|
||||
result = ph.prompt_format("p", name="Alice")
|
||||
assert result == "hello Alice"
|
||||
|
||||
|
||||
def test_flag_filter_unflagged_lines_always_kept():
|
||||
ph = PromptHandler()
|
||||
ph.load_prompt_dict({"p": "line1\nline2\nline3"})
|
||||
|
|
|
|||
|
|
@ -11,17 +11,17 @@ assert the union content and line order directly.
|
|||
from reme.schema import FileChunk
|
||||
from reme.steps.index._source_format import join_chunk_entries, merge_session_chunk_intervals, render_chunk_entries
|
||||
|
||||
_DIALOG_DIR = "session"
|
||||
_SESSION_DIR = "session"
|
||||
|
||||
|
||||
def _render(chunks: list[FileChunk], dialog_dir: str, **kwargs) -> str:
|
||||
def _render(chunks: list[FileChunk], session_dir: str, **kwargs) -> str:
|
||||
"""Merge session chunks, render entries, then join — mirroring the search steps."""
|
||||
return join_chunk_entries(
|
||||
render_chunk_entries(merge_session_chunk_intervals(chunks, dialog_dir), dialog_dir, **kwargs),
|
||||
render_chunk_entries(merge_session_chunk_intervals(chunks, session_dir), session_dir, **kwargs),
|
||||
)
|
||||
|
||||
|
||||
def _chunk(start: int, end: int, text: str, score: float = 1.0, path: str = "session/s1.jsonl") -> FileChunk:
|
||||
def _chunk(start: int, end: int, text: str, score: float = 1.0, path: str = "session/dialog/s1.jsonl") -> FileChunk:
|
||||
return FileChunk(path=path, start_line=start, end_line=end, text=text, scores={"score": score})
|
||||
|
||||
|
||||
|
|
@ -30,7 +30,7 @@ def test_overlapping_session_chunks_merge_into_union_without_duplicates():
|
|||
a = _chunk(1, 3, "m1\nm2\nm3\n")
|
||||
b = _chunk(3, 5, "m3\nm4\nm5\n")
|
||||
|
||||
answer = _render([a, b], _DIALOG_DIR, include_source=False)
|
||||
answer = _render([a, b], _SESSION_DIR, include_source=False)
|
||||
|
||||
assert answer == "m1\nm2\nm3\nm4\nm5"
|
||||
|
||||
|
|
@ -40,7 +40,7 @@ def test_contained_session_chunk_is_absorbed_by_the_larger_range():
|
|||
big = _chunk(1, 5, "m1\nm2\nm3\nm4\nm5\n")
|
||||
small = _chunk(2, 4, "m2\nm3\nm4\n")
|
||||
|
||||
answer = _render([big, small], _DIALOG_DIR, include_source=False)
|
||||
answer = _render([big, small], _SESSION_DIR, include_source=False)
|
||||
|
||||
assert answer == "m1\nm2\nm3\nm4\nm5"
|
||||
|
||||
|
|
@ -50,7 +50,7 @@ def test_adjacent_session_chunks_merge_end_plus_one_equals_next_start():
|
|||
a = _chunk(1, 3, "m1\nm2\nm3\n")
|
||||
b = _chunk(4, 6, "m4\nm5\nm6\n")
|
||||
|
||||
answer = _render([a, b], _DIALOG_DIR, include_source=False)
|
||||
answer = _render([a, b], _SESSION_DIR, include_source=False)
|
||||
|
||||
assert answer == "m1\nm2\nm3\nm4\nm5\nm6"
|
||||
|
||||
|
|
@ -60,17 +60,17 @@ def test_session_chunks_with_a_gap_are_not_merged():
|
|||
a = _chunk(1, 3, "m1\nm2\nm3\n")
|
||||
b = _chunk(5, 6, "m5\nm6\n") # line 4 missing -> not adjacent
|
||||
|
||||
answer = _render([a, b], _DIALOG_DIR, include_source=False)
|
||||
answer = _render([a, b], _SESSION_DIR, include_source=False)
|
||||
|
||||
assert answer == "m1\nm2\nm3\n\nm5\nm6"
|
||||
|
||||
|
||||
def test_session_chunks_from_different_files_are_not_merged():
|
||||
"""Overlapping ranges in different session files must stay separate."""
|
||||
a = _chunk(1, 3, "A1\nA2\nA3\n", path="session/s1.jsonl")
|
||||
b = _chunk(2, 4, "B2\nB3\nB4\n", path="session/s2.jsonl")
|
||||
a = _chunk(1, 3, "A1\nA2\nA3\n", path="session/dialog/s1.jsonl")
|
||||
b = _chunk(2, 4, "B2\nB3\nB4\n", path="session/dialog/s2.jsonl")
|
||||
|
||||
answer = _render([a, b], _DIALOG_DIR, include_source=False)
|
||||
answer = _render([a, b], _SESSION_DIR, include_source=False)
|
||||
|
||||
assert answer == "A1\nA2\nA3\n\nB2\nB3\nB4"
|
||||
|
||||
|
|
@ -80,7 +80,7 @@ def test_non_session_chunks_are_never_merged():
|
|||
a = _chunk(1, 3, "M1\nM2\nM3\n", path="daily/a.md")
|
||||
b = _chunk(2, 4, "M2\nM3\nM4\n", path="daily/a.md")
|
||||
|
||||
answer = _render([a, b], _DIALOG_DIR, include_source=False)
|
||||
answer = _render([a, b], _SESSION_DIR, include_source=False)
|
||||
|
||||
assert answer == "M1\nM2\nM3\n\nM2\nM3\nM4"
|
||||
|
||||
|
|
@ -91,7 +91,7 @@ def test_merge_preserves_line_order_regardless_of_input_rank_order():
|
|||
earlier = _chunk(1, 3, "m1\nm2\nm3\n", score=1.0)
|
||||
|
||||
# Higher-scored later-range chunk is listed first (as a ranker would).
|
||||
answer = _render([later, earlier], _DIALOG_DIR, include_source=False)
|
||||
answer = _render([later, earlier], _SESSION_DIR, include_source=False)
|
||||
|
||||
assert answer == "m1\nm2\nm3\nm4\nm5"
|
||||
|
||||
|
|
@ -101,10 +101,10 @@ def test_merged_header_spans_the_union_range_and_keeps_best_score():
|
|||
a = _chunk(1, 3, "m1\nm2\nm3\n", score=2.0)
|
||||
b = _chunk(3, 5, "m3\nm4\nm5\n", score=7.0)
|
||||
|
||||
answer = _render([a, b], _DIALOG_DIR, include_source=True)
|
||||
answer = _render([a, b], _SESSION_DIR, include_source=True)
|
||||
|
||||
assert answer.count("==========") == 2 # exactly one header (open + close markers)
|
||||
assert "session/s1.jsonl:1-5" in answer
|
||||
assert "session/dialog/s1.jsonl:1-5" in answer
|
||||
assert "score=7.0000" in answer
|
||||
|
||||
|
||||
|
|
@ -114,7 +114,7 @@ def test_separate_intervals_in_same_file_stay_separate():
|
|||
b = _chunk(3, 4, "m3\nm4\n") # adjacent to a -> merges with a into 1-4
|
||||
c = _chunk(10, 11, "m10\nm11\n") # far away -> separate
|
||||
|
||||
answer = _render([a, b, c], _DIALOG_DIR, include_source=False)
|
||||
answer = _render([a, b, c], _SESSION_DIR, include_source=False)
|
||||
|
||||
assert answer == "m1\nm2\nm3\nm4\n\nm10\nm11"
|
||||
|
||||
|
|
@ -123,12 +123,12 @@ def test_same_file_units_stay_adjacent_and_sorted_even_when_interleaved_by_rank(
|
|||
"""Two disjoint units of one session file are grouped together and ordered by
|
||||
``start_line``, even when a different file is ranked between them and the
|
||||
lower interval was ranked last."""
|
||||
s1_high = _chunk(10, 12, "S1x\nS1y\nS1z\n", score=9.0, path="session/s1.jsonl")
|
||||
s2_mid = _chunk(1, 3, "S2a\nS2b\nS2c\n", score=5.0, path="session/s2.jsonl")
|
||||
s1_low = _chunk(1, 3, "S1a\nS1b\nS1c\n", score=1.0, path="session/s1.jsonl")
|
||||
s1_high = _chunk(10, 12, "S1x\nS1y\nS1z\n", score=9.0, path="session/dialog/s1.jsonl")
|
||||
s2_mid = _chunk(1, 3, "S2a\nS2b\nS2c\n", score=5.0, path="session/dialog/s2.jsonl")
|
||||
s1_low = _chunk(1, 3, "S1a\nS1b\nS1c\n", score=1.0, path="session/dialog/s1.jsonl")
|
||||
|
||||
# Rank order (as a ranker would emit, by score desc): s1[10-12], s2[1-3], s1[1-3].
|
||||
answer = _render([s1_high, s2_mid, s1_low], _DIALOG_DIR, include_source=False)
|
||||
answer = _render([s1_high, s2_mid, s1_low], _SESSION_DIR, include_source=False)
|
||||
|
||||
# s1's two units are adjacent and sorted by start_line (1-3 before 10-12),
|
||||
# placed at s1's earliest rank (0), so the whole s1 block precedes s2.
|
||||
|
|
@ -137,13 +137,13 @@ def test_same_file_units_stay_adjacent_and_sorted_even_when_interleaved_by_rank(
|
|||
|
||||
def test_same_file_units_adjacency_with_source_headers():
|
||||
"""Header view: same-file units are contiguous and ascending; other files follow."""
|
||||
s1_high = _chunk(10, 12, "S1x\nS1y\nS1z\n", score=9.0, path="session/s1.jsonl")
|
||||
s2_mid = _chunk(1, 3, "S2a\nS2b\nS2c\n", score=5.0, path="session/s2.jsonl")
|
||||
s1_low = _chunk(1, 3, "S1a\nS1b\nS1c\n", score=1.0, path="session/s1.jsonl")
|
||||
s1_high = _chunk(10, 12, "S1x\nS1y\nS1z\n", score=9.0, path="session/dialog/s1.jsonl")
|
||||
s2_mid = _chunk(1, 3, "S2a\nS2b\nS2c\n", score=5.0, path="session/dialog/s2.jsonl")
|
||||
s1_low = _chunk(1, 3, "S1a\nS1b\nS1c\n", score=1.0, path="session/dialog/s1.jsonl")
|
||||
|
||||
answer = _render([s1_high, s2_mid, s1_low], _DIALOG_DIR, include_source=True)
|
||||
answer = _render([s1_high, s2_mid, s1_low], _SESSION_DIR, include_source=True)
|
||||
|
||||
headers = [line for line in answer.splitlines() if line.startswith("==========")]
|
||||
assert headers[0].split(" [")[0] == "========== session/s1.jsonl:1-3"
|
||||
assert headers[1].split(" [")[0] == "========== session/s1.jsonl:10-12"
|
||||
assert headers[2].split(" [")[0] == "========== session/s2.jsonl:1-3"
|
||||
assert headers[0].split(" [")[0] == "========== session/dialog/s1.jsonl:1-3"
|
||||
assert headers[1].split(" [")[0] == "========== session/dialog/s1.jsonl:10-12"
|
||||
assert headers[2].split(" [")[0] == "========== session/dialog/s2.jsonl:1-3"
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue