diff --git a/benchmark/README.md b/benchmark/README.md deleted file mode 100644 index 0522b09e..00000000 --- a/benchmark/README.md +++ /dev/null @@ -1,155 +0,0 @@ -[中文版 / Chinese version](./README_ZH.md) - -# ReMe Benchmarks - -Reproduction guide for the two memory benchmarks shipped with ReMe: - -- **LongMemEval** — long-term memory over multi-session chat histories. -- **BEAM** — memory capability over long-context chat cases with rubric-based judging. - -Each benchmark runs its own end-to-end pipeline: ingest sessions into an isolated -per-item workspace, answer probing questions via an agentic (ReAct) mode, -then score answers with an LLM-as-judge. - -## 1. Prerequisites - -Install ReMe with dev + core extras (Python 3.11+): - -```bash -pip install -e ".[dev,core]" -``` - -Configure model credentials in a project-root `.env` file (copied from `example.env`). -The runners auto-load `.env` from the repository root. Required variables typically include: - -```bash -LLM_API_KEY=... -LLM_BASE_URL=... -EMBEDDING_API_KEY=... -EMBEDDING_BASE_URL=... -``` - -Model names and component wiring live in the ReMe configs referenced by each benchmark -(`reme/config/lme.yaml` and `reme/config/beam.yaml`). - -## 2. Download Datasets - -Each benchmark keeps its own data under its directory: -`benchmark//dataset` (input data), `benchmark//workspaces` -(per-item memory workspaces), and `benchmark//results` (evaluation outputs). -All three are excluded from Git. - -**LongMemEval** — ReMe uses only the **cleaned-S** split, hosted on HuggingFace: -[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2) -(the script downloads via the hf-mirror.com mirror; to use a different mirror, -modify `BASE_URL` in `download.py`): - -```bash -cd benchmark/longmemeval -python download.py # saves dataset/longmemeval_s_reme_cleaned.json; skips if already present -``` - -**BEAM** (public repository, cloned into `benchmark/beam/dataset/`): - -```bash -mkdir -p benchmark/beam/dataset -cd benchmark/beam/dataset -git clone https://github.com/mohammadtavakoli78/BEAM.git -``` - -After cloning, `benchmark/beam/dataset/BEAM/` should contain `chats/`, `src/`, -`topics/` and other subdirectories. - -## 3. Run LongMemEval - -From the repository root: - -```bash -python benchmark/longmemeval/run.py -python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml -python benchmark/longmemeval/run.py -q # quiet: only eval-level logs -python benchmark/longmemeval/run.py --log-level WARNING # reduce eval runner logs -python benchmark/longmemeval/run.py --reme-log-level WARNING # reduce reme internal logs -python benchmark/longmemeval/run.py --eval_only # reuse existing workspaces, query + judge only -``` - -### Pipeline - -1. Load the dataset (ground truth is embedded in the data file). -2. For each item, create an isolated workspace and ingest sessions in chronological order. -3. Trigger `auto_dream` when consecutive sessions cross the configured hour (default 23:00). -4. Answer each question via agentic (ReAct) mode. -5. Judge the answer (binary yes/no) with the `answer_judge` job and print per-type accuracy. - -### Key config — `benchmark/longmemeval/config.yaml` - -| Key | Meaning | -| --- | --- | -| `dataset.path` | Dataset file to evaluate (e.g. `longmemeval_s_reme_cleaned.json`); ground truth is included. | -| `dataset.start_index` / `num_items` | Slice of items to evaluate. | -| `dataset.question_types` | Filter by question type; empty = all. | -| `dataset.workspace_root` | Per-item workspace root (`benchmark/longmemeval/workspaces/longmemeval-s`). | -| `evaluation.num_workers` | `0` = auto (cpu-2), `1` = sequential, `>1` = parallel. | -| `evaluation.filter_future_sessions` | Only ingest sessions with timestamp ≤ `question_date`. | -| `reme.config` | ReMe config used (`lme.yaml`). | -| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | Dream triggering behavior. | -| `output.dir` | Results directory (`benchmark/longmemeval/results`). | - -## 4. Run BEAM - -From the repository root: - -```bash -python benchmark/beam/run.py -python benchmark/beam/run.py --config benchmark/beam/config.yaml -python benchmark/beam/run.py -q # quiet -python benchmark/beam/run.py --eval_only # reuse existing workspaces, query + judge only -``` - -### Pipeline - -1. For each case, load `chat.json` and convert each batch into a ReMe session. -2. Ingest sessions in chronological order into an isolated workspace, then `digest_update`. -3. Answer each probing question via agentic (ReAct) mode. -4. Score answers with BEAM's rubric-based `answer_judge` job and print per-type averages. - -### Key config — `benchmark/beam/config.yaml` - -| Key | Meaning | -| --- | --- | -| `dataset.beam_root` | BEAM dataset root (`benchmark/beam/dataset/BEAM`). | -| `dataset.chat_size` | Variant to run: `100K` / `500K` / `1M` / `10M`. | -| `dataset.case_ids` | Specific cases (e.g. `["1","2"]`); empty = all cases. | -| `dataset.start_index` / `num_items` | Case pagination (`num_items` `0` = all). | -| `dataset.workspace_root` | Per-case workspace root (`benchmark/beam/workspaces/beam`). | -| `evaluation.num_workers` | `0` = auto, `1` = sequential, `>1` = parallel. | -| `reme.config` | ReMe config used (`beam.yaml`). | -| `output.dir` | Results directory (`benchmark/beam/results`). | - -## 5. Outputs & Logs - -- **Results**: JSON files written to `output.dir` - (`results_.json` for LongMemEval, - `results__.json` for BEAM). A summary with per-type - accuracy/score is also printed to the console. -- **Logs**: when `output.log_to_file` is enabled, per-run logs are written to - `logs/_/` (a `runner.log` plus one `worker-.log` - per worker process). - -## 6. Stopping a Run - -Parallel runs spawn a process tree. To terminate a run and all its workers cleanly: - -```bash -bash benchmark/kill.sh -``` - -The script gracefully sends `SIGTERM` to the whole process tree, then escalates to -`SIGKILL` for any process that does not exit within 5 seconds. - -## 7. Reference Results - -Recorded evaluation results are available in: - -- [`longmemeval.md`](./results_md/longmemeval.md) -- [`beam.md`](./results_md/beam.md) diff --git a/benchmark/README_ZH.md b/benchmark/README_ZH.md deleted file mode 100644 index 0ac8fd3e..00000000 --- a/benchmark/README_ZH.md +++ /dev/null @@ -1,149 +0,0 @@ -# ReMe 评测复现说明 - -ReMe 内置两个记忆能力评测基准的复现指南： - -- **LongMemEval** —— 面向多轮多会话历史的长期记忆能力评测。 -- **BEAM** —— 面向长上下文对话场景、基于评分细则（rubric）打分的记忆能力评测。 - -每个基准都会运行完整的端到端流程：将会话摄入独立的按条目隔离的工作区， -以 agentic（ReAct）模式回答探测问题， -最后由 LLM-as-judge 对答案进行打分。 - -## 1. 环境准备 - -安装 ReMe 及 dev + core 附加依赖（Python 3.11+）： - -```bash -pip install -e ".[dev,core]" -``` - -在项目根目录配置 `.env`（可从 `example.env` 复制），运行脚本会自动从仓库根目录加载 `.env`。 -通常需要以下变量： - -```bash -LLM_API_KEY=... -LLM_BASE_URL=... -EMBEDDING_API_KEY=... -EMBEDDING_BASE_URL=... -``` - -模型名称与组件装配位于各基准引用的 ReMe 配置中 -（`reme/config/lme.yaml` 与 `reme/config/beam.yaml`）。 - -## 2. 下载数据集 - -每个基准的数据都存放在各自目录下：`benchmark//dataset`（输入数据）、 -`benchmark//workspaces`（按条目隔离的记忆工作区）、 -`benchmark//results`（评测输出）。三者均不纳入 Git 版本管理。 - -**LongMemEval** —— ReMe 仅使用 **cleaned-S** 版本，数据托管在 HuggingFace： -[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2) -（下载脚本经 hf-mirror.com 镜像源获取，如需更换源请修改 `download.py` 中的 `BASE_URL`）： - -```bash -cd benchmark/longmemeval -python download.py # 保存为 dataset/longmemeval_s_reme_cleaned.json，已存在则自动跳过 -``` - -**BEAM**（公开仓库，clone 到 `benchmark/beam/dataset/` 下）： - -```bash -mkdir -p benchmark/beam/dataset -cd benchmark/beam/dataset -git clone https://github.com/mohammadtavakoli78/BEAM.git -``` - -clone 完成后，`benchmark/beam/dataset/BEAM/` 目录下应包含 `chats/`、`src/`、`topics/` 等子目录。 - -## 3. 运行 LongMemEval - -在仓库根目录执行： - -```bash -python benchmark/longmemeval/run.py -python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml -python benchmark/longmemeval/run.py -q # 安静模式：仅评测级日志 -python benchmark/longmemeval/run.py --log-level WARNING # 降低评测 runner 日志 -python benchmark/longmemeval/run.py --reme-log-level WARNING # 降低 reme 内部日志 -python benchmark/longmemeval/run.py --eval_only # 复用已有工作区，仅执行查询 + 评判 -``` - -### 流程 - -1. 加载数据集（ground truth 已内嵌在数据文件中）。 -2. 为每个条目创建独立工作区，按时间顺序摄入会话。 -3. 当相邻会话跨越配置的时刻（默认 23:00）时触发 `auto_dream`。 -4. 以 agentic（ReAct）模式回答每个问题。 -5. 通过 `answer_judge` 任务对答案做二元（yes/no）评判，并输出各类型准确率。 - -### 关键配置 —— `benchmark/longmemeval/config.yaml` - -| 配置项 | 含义 | -| --- | --- | -| `dataset.path` | 待评测的数据集文件（如 `longmemeval_s_reme_cleaned.json`），已包含 ground truth。 | -| `dataset.start_index` / `num_items` | 评测条目的切片范围。 | -| `dataset.question_types` | 按问题类型过滤，空表示全部。 | -| `dataset.workspace_root` | 条目工作区根目录（`benchmark/longmemeval/workspaces/longmemeval-s`）。 | -| `evaluation.num_workers` | `0` = 自动（cpu-2），`1` = 串行，`>1` = 并行。 | -| `evaluation.filter_future_sessions` | 仅摄入时间戳 ≤ `question_date` 的会话。 | -| `reme.config` | 使用的 ReMe 配置（`lme.yaml`）。 | -| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | dream 触发行为。 | -| `output.dir` | 结果目录（`benchmark/longmemeval/results`）。 | - -## 4. 运行 BEAM - -在仓库根目录执行： - -```bash -python benchmark/beam/run.py -python benchmark/beam/run.py --config benchmark/beam/config.yaml -python benchmark/beam/run.py -q # 安静模式 -python benchmark/beam/run.py --eval_only # 复用已有工作区，仅执行查询 + 评判 -``` - -### 流程 - -1. 为每个 case 加载 `chat.json`，将每个 batch 转换为一个 ReMe 会话。 -2. 按时间顺序将会话摄入独立工作区，随后执行 `digest_update`。 -3. 以 agentic（ReAct）模式回答每个探测问题。 -4. 通过 BEAM 基于 rubric 的 `answer_judge` 任务打分，并输出各类型平均分。 - -### 关键配置 —— `benchmark/beam/config.yaml` - -| 配置项 | 含义 | -| --- | --- | -| `dataset.beam_root` | BEAM 数据集根目录（`benchmark/beam/dataset/BEAM`）。 | -| `dataset.chat_size` | 运行的变体：`100K` / `500K` / `1M` / `10M`。 | -| `dataset.case_ids` | 指定 case（如 `["1","2"]`），空表示全部。 | -| `dataset.start_index` / `num_items` | case 分页（`num_items` 为 `0` 表示全部）。 | -| `dataset.workspace_root` | case 工作区根目录（`benchmark/beam/workspaces/beam`）。 | -| `evaluation.num_workers` | `0` = 自动，`1` = 串行，`>1` = 并行。 | -| `reme.config` | 使用的 ReMe 配置（`beam.yaml`）。 | -| `output.dir` | 结果目录（`benchmark/beam/results`）。 | - -## 5. 输出与日志 - -- **结果**：JSON 文件写入 `output.dir` - （LongMemEval 为 `results_.json`， - BEAM 为 `results__.json`）。同时控制台会打印含各类型 - 准确率/分数的汇总。 -- **日志**：当 `output.log_to_file` 开启时，每次运行的日志写入 - `logs/_/`（包含一个 `runner.log` 及每个 worker 进程的 - `worker-.log`）。 - -## 6. 终止运行 - -并行运行会派生进程树。若要干净地终止某次运行及其全部 worker： - -```bash -bash benchmark/kill.sh -``` - -该脚本会先向整个进程树发送 `SIGTERM` 优雅终止，对 5 秒内未退出的进程再升级为 `SIGKILL`。 - -## 7. 参考结果 - -已记录的评测结果见： - -- [`longmemeval.md`](./results_md/longmemeval.md) -- [`beam.md`](./results_md/beam.md) diff --git a/benchmark/beam/README.md b/benchmark/beam/README.md new file mode 100644 index 00000000..aef8a3b7 --- /dev/null +++ b/benchmark/beam/README.md @@ -0,0 +1,124 @@ +[中文版 / Chinese version](./README_ZH.md) + +# BEAM Benchmark + +BEAM is a benchmark for **memory capability over long-context chat cases**. Each +case contains a very long chat history split into batches; ReMe converts each +batch into a session, ingests them in chronological order, then answers probing +questions via an agentic (ReAct) mode. Answers are scored with BEAM's +rubric-based `answer_judge` job, which produces both a graded score and a binary +verdict, and per-type averages are reported. + +BEAM ships dataset variants by chat size — `100K` / `500K` / `1M` / `10M` — so +memory systems can be stressed at different context lengths. Question types +include abstention, contradiction resolution, event ordering, information +extraction, instruction following, knowledge update, multi-session reasoning, +preference following, summarization, and temporal reasoning. + +> For the shared setup (dependencies, credentials, log conventions) see the +> [top-level benchmark README](../README.md). + +## 1. Get the Dataset + +BEAM is a public repository, cloned into `benchmark/beam/dataset/`: + +```bash +mkdir -p benchmark/beam/dataset +cd benchmark/beam/dataset +git clone https://github.com/mohammadtavakoli78/BEAM.git +``` + +After cloning, `benchmark/beam/dataset/BEAM/` should contain `chats/`, `src/`, +`topics/` and other subdirectories. + +## 2. Run + +From the repository root: + +```bash +python benchmark/beam/run.py +python benchmark/beam/run.py --config benchmark/beam/config.yaml +python benchmark/beam/run.py -q # quiet +python benchmark/beam/run.py --eval_only # reuse existing workspaces, query + judge only +``` + +## 3. Pipeline + +1. For each case, load `chat.json` and convert each batch into a ReMe session. +2. Ingest sessions in chronological order into an isolated workspace, then `digest_update`. +3. Answer each probing question via agentic (ReAct) mode. +4. Score answers with BEAM's rubric-based `answer_judge` job and print per-type averages. + +## 4. Key config — `benchmark/beam/config.yaml` + +| Key | Meaning | +| --- | --- | +| `dataset.beam_root` | BEAM dataset root (`benchmark/beam/dataset/BEAM`). | +| `dataset.chat_size` | Variant to run: `100K` / `500K` / `1M` / `10M`. | +| `dataset.case_ids` | Specific cases (e.g. `["1","2"]`); empty = all cases. | +| `dataset.start_index` / `num_items` | Case pagination (`num_items` `0` = all). | +| `dataset.workspace_root` | Per-case workspace root (`benchmark/beam/workspaces/beam`). | +| `evaluation.num_workers` | `0` = auto, `1` = sequential, `>1` = parallel. | +| `reme.config` | ReMe config used (`beam.yaml`). | +| `output.dir` | Results directory (`benchmark/beam/results`). | + +## 5. Outputs + +Results are JSON files written to `output.dir` as +`results__.json`, with a per-type score summary also +printed to the console. Logging conventions are shared across benchmarks — see +the [top-level README](../README.md#outputs--logs). + +## 6. Reference Results + +> The results below use the longmemeval-version prompt. + +### 100K + +agentscope==2.0.4.post1, conda reme env, 20 workers, eval-only (reusing prebuilt memory) +(2026-08-05, 20 cases / 400 Qs, total 46.0 min) + +| Type | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q | +|---|---|---|---|---|---|---| +| abstention | 0.550 | 0.550 | 96,031 | 1,070 | 97,101 | 4.58 | +| contradiction_resolution | 0.438 | 0.412 | 32,263 | 872 | 33,135 | 2.48 | +| event_ordering | 0.501 | 0.423 | 140,195 | 5,163 | 145,358 | 4.70 | +| information_extraction | 0.873 | 0.832 | 50,245 | 883 | 51,128 | 3.15 | +| instruction_following | 0.750 | 0.725 | 37,986 | 848 | 38,834 | 2.67 | +| knowledge_update | 0.688 | 0.675 | 31,198 | 651 | 31,849 | 2.27 | +| multi_session_reasoning | 0.626 | 0.584 | 85,038 | 4,563 | 89,601 | 4.28 | +| preference_following | 0.925 | 0.912 | 34,281 | 989 | 35,270 | 2.50 | +| summarization | 0.623 | 0.461 | 89,657 | 2,056 | 91,713 | 4.12 | +| temporal_reasoning | 0.637 | 0.625 | 34,563 | 1,049 | 35,612 | 2.52 | +| **OVERALL** | **0.661** | **0.620** | **63,146** | **1,814** | **64,960** | **3.33** | + +Memory Construction average token consumption (default agent, full build over 20 cases): + +| Agent | input tok/case | output tok/case | total tok/case | +|---|---|---|---| +| default | 2,172,316 | 136,697 | 2,309,013 | + +### 1M + +agentscope==2.0.4.post1, conda reme env, 20 workers, full memory build +(2026-08-05, 35 cases / 700 Qs, total 459.2 min) + +| Type | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q | +|---|---|---|---|---|---|---| +| abstention | 0.429 | 0.429 | 118,707 | 1,178 | 119,886 | 4.20 | +| contradiction_resolution | 0.391 | 0.364 | 49,787 | 810 | 50,597 | 2.50 | +| event_ordering | 0.558 | 0.456 | 201,514 | 3,889 | 205,403 | 4.79 | +| information_extraction | 0.809 | 0.772 | 78,950 | 894 | 79,844 | 3.00 | +| instruction_following | 0.852 | 0.832 | 55,757 | 924 | 56,681 | 2.81 | +| knowledge_update | 0.779 | 0.771 | 45,981 | 665 | 46,646 | 2.37 | +| multi_session_reasoning | 0.658 | 0.612 | 138,133 | 2,873 | 141,006 | 4.40 | +| preference_following | 0.798 | 0.777 | 51,796 | 920 | 52,716 | 2.53 | +| summarization | 0.693 | 0.537 | 158,794 | 2,905 | 161,700 | 4.44 | +| temporal_reasoning | 0.536 | 0.536 | 100,176 | 3,148 | 103,324 | 3.90 | +| **OVERALL** | **0.650** | **0.609** | **99,959** | **1,821** | **101,780** | **3.49** | + +Memory Construction average token consumption (default agent, full build over 35 cases): + +| Agent | input tok/case | output tok/case | total tok/case | +|---|---|---|---| +| default | 31,943,817 | 1,417,061 | 33,360,878 | diff --git a/benchmark/beam/README_ZH.md b/benchmark/beam/README_ZH.md new file mode 100644 index 00000000..f562193b --- /dev/null +++ b/benchmark/beam/README_ZH.md @@ -0,0 +1,119 @@ +# BEAM 评测 + +[English version](./README.md) + +BEAM 是一个面向**长上下文对话场景**的记忆能力评测基准。每个 case 包含一段被切分为多个 +batch 的超长对话；ReMe 将每个 batch 转换为一个会话，按时间顺序摄入后，以 agentic（ReAct） +模式回答探测问题。答案由 BEAM 基于 rubric 的 `answer_judge` 任务打分，同时给出分级分数与二元 +判定，并输出各类型平均分。 + +BEAM 按对话规模提供多种数据变体 —— `100K` / `500K` / `1M` / `10M`，可在不同上下文长度下 +压测记忆系统。题型包括 abstention（拒答）、contradiction resolution（矛盾消解）、event +ordering（事件排序）、information extraction（信息抽取）、instruction following（指令遵循）、 +knowledge update（知识更新）、multi-session reasoning（多会话推理）、preference following +（偏好遵循）、summarization（摘要）与 temporal reasoning（时间推理）。 + +> 公共设置（依赖、凭据、日志约定）见[总评测说明](../README_ZH.md)。 + +## 1. 获取数据集 + +BEAM 是公开仓库，clone 到 `benchmark/beam/dataset/` 下： + +```bash +mkdir -p benchmark/beam/dataset +cd benchmark/beam/dataset +git clone https://github.com/mohammadtavakoli78/BEAM.git +``` + +clone 完成后，`benchmark/beam/dataset/BEAM/` 目录下应包含 `chats/`、`src/`、`topics/` 等子目录。 + +## 2. 运行 + +在仓库根目录执行： + +```bash +python benchmark/beam/run.py +python benchmark/beam/run.py --config benchmark/beam/config.yaml +python benchmark/beam/run.py -q # 安静模式 +python benchmark/beam/run.py --eval_only # 复用已有工作区，仅执行查询 + 评判 +``` + +## 3. 流程 + +1. 为每个 case 加载 `chat.json`，将每个 batch 转换为一个 ReMe 会话。 +2. 按时间顺序将会话摄入独立工作区，随后执行 `digest_update`。 +3. 以 agentic（ReAct）模式回答每个探测问题。 +4. 通过 BEAM 基于 rubric 的 `answer_judge` 任务打分，并输出各类型平均分。 + +## 4. 关键配置 —— `benchmark/beam/config.yaml` + +| 配置项 | 含义 | +| --- | --- | +| `dataset.beam_root` | BEAM 数据集根目录（`benchmark/beam/dataset/BEAM`）。 | +| `dataset.chat_size` | 运行的变体：`100K` / `500K` / `1M` / `10M`。 | +| `dataset.case_ids` | 指定 case（如 `["1","2"]`），空表示全部。 | +| `dataset.start_index` / `num_items` | case 分页（`num_items` 为 `0` 表示全部）。 | +| `dataset.workspace_root` | case 工作区根目录（`benchmark/beam/workspaces/beam`）。 | +| `evaluation.num_workers` | `0` = 自动，`1` = 串行，`>1` = 并行。 | +| `reme.config` | 使用的 ReMe 配置（`beam.yaml`）。 | +| `output.dir` | 结果目录（`benchmark/beam/results`）。 | + +## 5. 输出 + +结果以 JSON 文件写入 `output.dir`，文件名为 `results__.json`， +同时控制台会打印含各类型分数的汇总。日志约定在各基准间通用，见 +[总说明](../README_ZH.md#输出与日志)。 + +## 6. 参考结果 + +> 以下结果使用 longmemeval 版本的 prompt。 + +### 100K + +agentscope==2.0.4.post1，conda reme 环境，20 并发，eval-only（复用已构建 memory） +（2026-08-05，20 cases / 400 Qs，总耗时 46.0 min） + +| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q | +|---|---|---|---|---|---|---| +| abstention | 0.550 | 0.550 | 96,031 | 1,070 | 97,101 | 4.58 | +| contradiction_resolution | 0.438 | 0.412 | 32,263 | 872 | 33,135 | 2.48 | +| event_ordering | 0.501 | 0.423 | 140,195 | 5,163 | 145,358 | 4.70 | +| information_extraction | 0.873 | 0.832 | 50,245 | 883 | 51,128 | 3.15 | +| instruction_following | 0.750 | 0.725 | 37,986 | 848 | 38,834 | 2.67 | +| knowledge_update | 0.688 | 0.675 | 31,198 | 651 | 31,849 | 2.27 | +| multi_session_reasoning | 0.626 | 0.584 | 85,038 | 4,563 | 89,601 | 4.28 | +| preference_following | 0.925 | 0.912 | 34,281 | 989 | 35,270 | 2.50 | +| summarization | 0.623 | 0.461 | 89,657 | 2,056 | 91,713 | 4.12 | +| temporal_reasoning | 0.637 | 0.625 | 34,563 | 1,049 | 35,612 | 2.52 | +| **OVERALL** | **0.661** | **0.620** | **63,146** | **1,814** | **64,960** | **3.33** | + +Memory Construction 平均 token 消耗（default agent，20 cases 全量构建）： + +| Agent | input tok/case | output tok/case | total tok/case | +|---|---|---|---| +| default | 2,172,316 | 136,697 | 2,309,013 | + +### 1M + +agentscope==2.0.4.post1，conda reme 环境，20 并发，全量构建 memory +（2026-08-05，35 cases / 700 Qs，总耗时 459.2 min） + +| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q | +|---|---|---|---|---|---|---| +| abstention | 0.429 | 0.429 | 118,707 | 1,178 | 119,886 | 4.20 | +| contradiction_resolution | 0.391 | 0.364 | 49,787 | 810 | 50,597 | 2.50 | +| event_ordering | 0.558 | 0.456 | 201,514 | 3,889 | 205,403 | 4.79 | +| information_extraction | 0.809 | 0.772 | 78,950 | 894 | 79,844 | 3.00 | +| instruction_following | 0.852 | 0.832 | 55,757 | 924 | 56,681 | 2.81 | +| knowledge_update | 0.779 | 0.771 | 45,981 | 665 | 46,646 | 2.37 | +| multi_session_reasoning | 0.658 | 0.612 | 138,133 | 2,873 | 141,006 | 4.40 | +| preference_following | 0.798 | 0.777 | 51,796 | 920 | 52,716 | 2.53 | +| summarization | 0.693 | 0.537 | 158,794 | 2,905 | 161,700 | 4.44 | +| temporal_reasoning | 0.536 | 0.536 | 100,176 | 3,148 | 103,324 | 3.90 | +| **OVERALL** | **0.650** | **0.609** | **99,959** | **1,821** | **101,780** | **3.49** | + +Memory Construction 平均 token 消耗（default agent，35 cases 全量构建）： + +| Agent | input tok/case | output tok/case | total tok/case | +|---|---|---|---| +| default | 31,943,817 | 1,417,061 | 33,360,878 | diff --git a/benchmark/kill.sh b/benchmark/beam/kill.sh similarity index 100% rename from benchmark/kill.sh rename to benchmark/beam/kill.sh diff --git a/benchmark/longmemeval/README.md b/benchmark/longmemeval/README.md new file mode 100644 index 00000000..070f921e --- /dev/null +++ b/benchmark/longmemeval/README.md @@ -0,0 +1,96 @@ +[中文版 / Chinese version](./README_ZH.md) + +# LongMemEval Benchmark + +LongMemEval is a benchmark for **long-term memory over multi-session chat +histories**. Each item provides a chronologically ordered set of chat sessions +between a user and an assistant, followed by a probing question whose answer is +only recoverable by reasoning over the user-owned memory. ReMe ingests the +sessions into an isolated per-item workspace, answers the question via an +agentic (ReAct) mode, and scores the answer with an LLM-as-judge. + +Question types include single-session (user / assistant / preference), +multi-session reasoning, knowledge update, and temporal reasoning. + +> For the shared setup (dependencies, credentials, log conventions) see the +> [top-level benchmark README](../README.md). + +## 1. Get the Dataset + +ReMe uses only the **cleaned-S** split, hosted on HuggingFace: +[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2). +The download script fetches it via the hf-mirror.com mirror; to use a different +mirror, modify `BASE_URL` in [`download.py`](./download.py). + +```bash +cd benchmark/longmemeval +python download.py # saves dataset/longmemeval_s_reme_cleaned.json; skips if already present +``` + +Ground truth is embedded in the data file. + +## 2. Run + +From the repository root: + +```bash +python benchmark/longmemeval/run.py +python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml +python benchmark/longmemeval/run.py -q # quiet: only eval-level logs +python benchmark/longmemeval/run.py --log-level WARNING # reduce eval runner logs +python benchmark/longmemeval/run.py --reme-log-level WARNING # reduce reme internal logs +python benchmark/longmemeval/run.py --eval_only # reuse existing workspaces, query + judge only +``` + +## 3. Pipeline + +1. Load the dataset (ground truth is embedded in the data file). +2. For each item, create an isolated workspace and ingest sessions in chronological order. +3. Trigger `auto_dream` when consecutive sessions cross the configured hour (default 23:00). +4. Answer each question via agentic (ReAct) mode. +5. Judge the answer (binary yes/no) with the `answer_judge` job and print per-type accuracy. + +## 4. Key config — `benchmark/longmemeval/config.yaml` + +| Key | Meaning | +| --- | --- | +| `dataset.path` | Dataset file to evaluate (e.g. `longmemeval_s_reme_cleaned.json`); ground truth is included. | +| `dataset.start_index` / `num_items` | Slice of items to evaluate. | +| `dataset.question_types` | Filter by question type; empty = all. | +| `dataset.workspace_root` | Per-item workspace root (`benchmark/longmemeval/workspaces/longmemeval-s`). | +| `evaluation.num_workers` | `0` = auto (cpu-2), `1` = sequential, `>1` = parallel. | +| `evaluation.filter_future_sessions` | Only ingest sessions with timestamp ≤ `question_date`. | +| `reme.config` | ReMe config used (`lme.yaml`). | +| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | Dream triggering behavior. | +| `output.dir` | Results directory (`benchmark/longmemeval/results`). | + +## 5. Outputs + +Results are JSON files written to `output.dir` as `results_.json`, +with a per-type accuracy summary also printed to the console. Logging +conventions are shared across benchmarks — see the +[top-level README](../README.md#outputs--logs). + +## 6. Reference Results + +### cleaned-s + +**Basic settings** + +1. Modified auto-memory prompt, auto-dream disabled. +2. All sessions in reme-memory are strictly earlier than the question time. + +**Results** + +agentscope==2.0.4.post1, conda reme env, 32 workers, eval-only (reusing prebuilt memory) +(2026-08-06, 500 items, total 10.0 min) + +| Type | Agentic | input tok/q | output tok/q | total tok/q | tool calls/q | +|---|---|---|---|---|---| +| knowledge-update | 0.910 | 31,581 | 589 | 32,169 | 2.90 | +| multi-session | 0.842 | 52,837 | 1,474 | 54,311 | 4.21 | +| single-session-assistant | 1.000 | 15,596 | 279 | 15,875 | 1.89 | +| single-session-preference | 0.633 | 36,802 | 818 | 37,620 | 3.60 | +| single-session-user | 0.986 | 27,433 | 359 | 27,792 | 2.60 | +| temporal-reasoning | 0.902 | 62,674 | 985 | 63,659 | 4.97 | +| **OVERALL** | **0.894** | **43,448** | **876** | **44,324** | **3.69** | \ No newline at end of file diff --git a/benchmark/longmemeval/README_ZH.md b/benchmark/longmemeval/README_ZH.md new file mode 100644 index 00000000..9c07b175 --- /dev/null +++ b/benchmark/longmemeval/README_ZH.md @@ -0,0 +1,90 @@ +# LongMemEval 评测 + +[English version](./README.md) + +LongMemEval 是一个面向**多轮多会话历史的长期记忆能力**的评测基准。每个条目提供一组按时间 +顺序排列的用户与助手之间的会话，以及一个只能通过推理用户自有记忆才能回答的探测问题。ReMe +将会话摄入按条目隔离的工作区，以 agentic（ReAct）模式回答问题，最后由 LLM-as-judge 打分。 + +题型包括单会话（user / assistant / preference）、多会话推理、知识更新与时间推理等。 + +> 公共设置（依赖、凭据、日志约定）见[总评测说明](../README_ZH.md)。 + +## 1. 获取数据集 + +ReMe 仅使用 **cleaned-S** 版本，数据托管在 HuggingFace： +[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)。 +下载脚本经 hf-mirror.com 镜像源获取，如需更换源请修改 [`download.py`](./download.py) 中的 +`BASE_URL`。 + +```bash +cd benchmark/longmemeval +python download.py # 保存为 dataset/longmemeval_s_reme_cleaned.json，已存在则自动跳过 +``` + +ground truth 已内嵌在数据文件中。 + +## 2. 运行 + +在仓库根目录执行： + +```bash +python benchmark/longmemeval/run.py +python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml +python benchmark/longmemeval/run.py -q # 安静模式：仅评测级日志 +python benchmark/longmemeval/run.py --log-level WARNING # 降低评测 runner 日志 +python benchmark/longmemeval/run.py --reme-log-level WARNING # 降低 reme 内部日志 +python benchmark/longmemeval/run.py --eval_only # 复用已有工作区，仅执行查询 + 评判 +``` + +## 3. 流程 + +1. 加载数据集（ground truth 已内嵌在数据文件中）。 +2. 为每个条目创建独立工作区，按时间顺序摄入会话。 +3. 当相邻会话跨越配置的时刻（默认 23:00）时触发 `auto_dream`。 +4. 以 agentic（ReAct）模式回答每个问题。 +5. 通过 `answer_judge` 任务对答案做二元（yes/no）评判，并输出各类型准确率。 + +## 4. 关键配置 —— `benchmark/longmemeval/config.yaml` + +| 配置项 | 含义 | +| --- | --- | +| `dataset.path` | 待评测的数据集文件（如 `longmemeval_s_reme_cleaned.json`），已包含 ground truth。 | +| `dataset.start_index` / `num_items` | 评测条目的切片范围。 | +| `dataset.question_types` | 按问题类型过滤，空表示全部。 | +| `dataset.workspace_root` | 条目工作区根目录（`benchmark/longmemeval/workspaces/longmemeval-s`）。 | +| `evaluation.num_workers` | `0` = 自动（cpu-2），`1` = 串行，`>1` = 并行。 | +| `evaluation.filter_future_sessions` | 仅摄入时间戳 ≤ `question_date` 的会话。 | +| `reme.config` | 使用的 ReMe 配置（`lme.yaml`）。 | +| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | dream 触发行为。 | +| `output.dir` | 结果目录（`benchmark/longmemeval/results`）。 | + +## 5. 输出 + +结果以 JSON 文件写入 `output.dir`，文件名为 `results_.json`， +同时控制台会打印含各类型准确率的汇总。日志约定在各基准间通用，见 +[总说明](../README_ZH.md#输出与日志)。 + +## 6. 参考结果 + +### cleaned-s + +**基础设置** + +1. 使用修改后的 auto-memory prompt，关闭 auto-dream 机制 +2. reme-memory 中的全部 session 的时间一定早于 question 的时间 + +**结果** + +agentscope==2.0.4.post1, conda reme env, 32 workers, eval-only（复用预构建记忆） +（2026-08-06，500 题，总计 10.0 min） + +| 类型 | Agentic | input tok/q | output tok/q | total tok/q | tool calls/q | +|---|---|---|---|---|---| +| knowledge-update | 0.910 | 31,581 | 589 | 32,169 | 2.90 | +| multi-session | 0.842 | 52,837 | 1,474 | 54,311 | 4.21 | +| single-session-assistant | 1.000 | 15,596 | 279 | 15,875 | 1.89 | +| single-session-preference | 0.633 | 36,802 | 818 | 37,620 | 3.60 | +| single-session-user | 0.986 | 27,433 | 359 | 27,792 | 2.60 | +| temporal-reasoning | 0.902 | 62,674 | 985 | 63,659 | 4.97 | +| **OVERALL** | **0.894** | **43,448** | **876** | **44,324** | **3.69** | diff --git a/benchmark/longmemeval/kill.sh b/benchmark/longmemeval/kill.sh new file mode 100644 index 00000000..9f4a5237 --- /dev/null +++ b/benchmark/longmemeval/kill.sh @@ -0,0 +1,76 @@ +#!/bin/bash +# 杀死指定进程及其所有子进程 +# Usage: bash kill.sh + +if [ -z "$1" ]; then + echo "Usage: bash kill.sh " + echo " 杀死指定进程及其所有子进程" + exit 1 +fi + +PID=$1 + +# 检查进程是否存在 +if ! kill -0 "$PID" 2>/dev/null; then + echo "进程 $PID 不存在" + exit 1 +fi + +# 递归收集所有子进程（包括子进程的子进程） +collect_children() { + local parent=$1 + local children + children=$(ps -o pid= --ppid "$parent" 2>/dev/null | tr -d ' ') + for child in $children; do + collect_children "$child" + done + echo "$parent" +} + +# 收集进程树（子进程在前，父进程在后，保证先杀子再杀父） +PROCESS_TREE=$(collect_children "$PID") +TOTAL=$(echo "$PROCESS_TREE" | wc -l | tr -d ' ') + +echo "进程树（共 $TOTAL 个进程）:" +while read -r p; do + cmd=$(ps -o args= -p "$p" 2>/dev/null | head -c 80) + printf " PID=%-8s %s\n" "$p" "$cmd" +done <<< "$PROCESS_TREE" + +# 先 SIGTERM 优雅终止 +echo "" +echo "发送 SIGTERM..." +while read -r p; do + kill "$p" 2>/dev/null +done <<< "$PROCESS_TREE" + +# 等待最多 5 秒 +for i in $(seq 1 5); do + alive=false + while read -r p; do + if kill -0 "$p" 2>/dev/null; then + alive=true + fi + done <<< "$PROCESS_TREE" + if [ "$alive" = false ]; then + break + fi + sleep 1 +done + +# 检查是否还有残留，强制 SIGKILL +remaining=false +while read -r p; do + if kill -0 "$p" 2>/dev/null; then + remaining=true + fi +done <<< "$PROCESS_TREE" + +if [ "$remaining" = true ]; then + echo "部分进程未响应，发送 SIGKILL..." + while read -r p; do + kill -9 "$p" 2>/dev/null + done <<< "$PROCESS_TREE" +fi + +echo "已终止进程树（根 PID=$PID，共 $TOTAL 个进程）" diff --git a/benchmark/results_md/beam.md b/benchmark/results_md/beam.md deleted file mode 100644 index 651690d7..00000000 --- a/benchmark/results_md/beam.md +++ /dev/null @@ -1,54 +0,0 @@ -# beam result - -## longmemeval版本的prompt - -### 100K - - -agentscope==2.0.4.post1, conda reme 环境, 20 并发, eval-only（复用已构建 memory） -(2026-08-05, 20 cases / 400 Qs, 总耗时 46.0 min) - -| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q | -|---|---|---|---|---|---|---| -| abstention | 0.550 | 0.550 | 96,031 | 1,070 | 97,101 | 4.58 | -| contradiction_resolution | 0.438 | 0.412 | 32,263 | 872 | 33,135 | 2.48 | -| event_ordering | 0.501 | 0.423 | 140,195 | 5,163 | 145,358 | 4.70 | -| information_extraction | 0.873 | 0.832 | 50,245 | 883 | 51,128 | 3.15 | -| instruction_following | 0.750 | 0.725 | 37,986 | 848 | 38,834 | 2.67 | -| knowledge_update | 0.688 | 0.675 | 31,198 | 651 | 31,849 | 2.27 | -| multi_session_reasoning | 0.626 | 0.584 | 85,038 | 4,563 | 89,601 | 4.28 | -| preference_following | 0.925 | 0.912 | 34,281 | 989 | 35,270 | 2.50 | -| summarization | 0.623 | 0.461 | 89,657 | 2,056 | 91,713 | 4.12 | -| temporal_reasoning | 0.637 | 0.625 | 34,563 | 1,049 | 35,612 | 2.52 | -| **OVERALL** | **0.661** | **0.620** | **63,146** | **1,814** | **64,960** | **3.33** | - -Memory Construction 平均 token 消耗（default agent, 20 cases 全量构建）: - -| Agent | input tok/case | output tok/case | total tok/case | -|---|---|---|---| -| default | 2,172,316 | 136,697 | 2,309,013 | - -### 1M - -agentscope==2.0.4.post1, conda reme 环境, 20 并发, 全量构建 memory -(2026-08-05, 35 cases / 700 Qs, 总耗时 459.2 min) - -| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q | -|---|---|---|---|---|---|---| -| abstention | 0.429 | 0.429 | 118,707 | 1,178 | 119,886 | 4.20 | -| contradiction_resolution | 0.391 | 0.364 | 49,787 | 810 | 50,597 | 2.50 | -| event_ordering | 0.558 | 0.456 | 201,514 | 3,889 | 205,403 | 4.79 | -| information_extraction | 0.809 | 0.772 | 78,950 | 894 | 79,844 | 3.00 | -| instruction_following | 0.852 | 0.832 | 55,757 | 924 | 56,681 | 2.81 | -| knowledge_update | 0.779 | 0.771 | 45,981 | 665 | 46,646 | 2.37 | -| multi_session_reasoning | 0.658 | 0.612 | 138,133 | 2,873 | 141,006 | 4.40 | -| preference_following | 0.798 | 0.777 | 51,796 | 920 | 52,716 | 2.53 | -| summarization | 0.693 | 0.537 | 158,794 | 2,905 | 161,700 | 4.44 | -| temporal_reasoning | 0.536 | 0.536 | 100,176 | 3,148 | 103,324 | 3.90 | -| **OVERALL** | **0.650** | **0.609** | **99,959** | **1,821** | **101,780** | **3.49** | - -Memory Construction 平均 token 消耗（default agent, 35 cases 全量构建）: - -| Agent | input tok/case | output tok/case | total tok/case | -|---|---|---|---| -| default | 31,943,817 | 1,417,061 | 33,360,878 | \ No newline at end of file diff --git a/benchmark/results_md/longmemeval.md b/benchmark/results_md/longmemeval.md deleted file mode 100644 index 59671395..00000000 --- a/benchmark/results_md/longmemeval.md +++ /dev/null @@ -1,92 +0,0 @@ -# LongMemEval 数据集测试结果 - -## cleaned-s - -**basic settings** - -1. 使用修改后的auto-memory prompt，关闭auto-dream机制 -2. reme-memory中的全部session的时间一定早于question的时间 - -**results ** - -1. Agentic answer框架回答，每次最多调用5次search - -| Category | Total | Correct | Wrong | Accuracy | -|---|---|---|---|---| -| single-session-user | 70 | 66 | 4 | 94.3% | -| single-session-assistant | 56 | 52 | 4 | 92.9% | -| knowledge-update | 78 | 60 | 18 | 76.9% | -| multi-session | 133 | 93 | 40 | 69.9% | -| temporal-reasoning | 133 | 78 | 55 | 58.6% | -| single-session-preference | 30 | 8 | 22 | 26.7% | -| **Overall** | **500** | **357** | **143** | **71.4%** | - -2. prompted-based amswer，每次固定使用原始query召回10个fileChunk - -| Category | Total | Correct | Wrong | Accuracy | -|---|---|---|---|---| -| single-session-assistant | 56 | 56 | 0 | 100.0% | -| single-session-user | 70 | 67 | 3 | 95.7% | -| knowledge-update | 78 | 69 | 9 | 88.5% | -| multi-session | 133 | 99 | 34 | 74.4% | -| temporal-reasoning | 133 | 83 | 50 | 62.4% | -| single-session-preference | 30 | 16 | 14 | 53.3% | -| **Overall** | **500** | **390** | **110** | **78.0%** | - -3. golden session。 使用与prompt-based answer相似的方法，唯一区别是，输入的chunk是longMemEval提供的golden session。 - -| Category | Total | Correct | Wrong | Accuracy | -|---|---|---|---|---| -| single-session-assistant | 56 | 56 | 0 | 100.0% | -| single-session-user | 70 | 69 | 1 | 98.6% | -| knowledge-update | 78 | 74 | 4 | 94.9% | -| temporal-reasoning | 133 | 124 | 9 | 93.2% | -| multi-session | 133 | 117 | 16 | 88.0% | -| single-session-preference | 30 | 17 | 13 | 56.7% | -| **Overall** | **500** | **457** | **43** | **91.4%** | - -4. golden session + time filter. 和上面一个实验的区别是，输入的golden被过滤了一次，要求输入session的时间戳必须早于question的时间才行。 - -一共被过滤掉了75个session，44个question受到了影响。temperal reasoning类型受影响最大。有20个case不包含任何一个groundtruth session。 根据golden session回答正确并且golden session非空，一共有424个case。 - -| Category | Total | Correct | Wrong | Accuracy | -|---|---|---|---|---| -| knowledge-update | 78 | 75 | 3 | 96.2% | -| single-session-user | 70 | 67 | 3 | 95.7% | -| multi-session | 133 | 122 | 11 | 91.7% | -| single-session-assistant | 56 | 55 | 1 | 98.2% | -| temporal-reasoning | 133 | 91 | 42 | 68.4% | -| single-session-preference | 30 | 16 | 14 | 53.3% | -| **Overall** | **500** | **426** | **74** | **85.2%** | - -5. 关闭auto-memory机制，根据原始query一次性混合检索召回原始session，计算recall. - -| Category | Total | yes-judge | recall@5 / yes | recall@10 / yes | -|---|---|---|---|---| -| knowledge-update | 78 | 75 | 99.3% | 100% | -| single-session-user | 70 | 67 | 100% | 100% | -| multi-session | 133 | 122 | 91.8% | 95.8% | -| single-session-assistant | 56 | 55 | 100% | 100% | -| temporal-reasoning | 133 | 91 | 87.6% | 94.2% | -| single-session-preference | 30 | 16 | 100% | 100% | -| **Overall** | **500** | **426** | **87.6%** | **94.2%** | - - -## 最终groundtruth - -### agentic + prompted（最终GT，2026-07-16） - - -| Category | Total | Agentic | Prompted limit=15 | -|---|---|---|---| -| single-session-assistant | 56 | 56/56 (100.0%) | 54/56 (96.4%) | -| single-session-user | 70 | 66/70 (94.3%) | 62/70 (88.6%) | -| knowledge-update | 78 | 75/78 (96.2%) | 67/78 (85.9%) | -| temporal-reasoning | 133 | 122/133 (91.7%) | 117/133 (88.0%) | -| multi-session | 133 | 115/133 (86.5%) | 101/133 (75.9%) | -| single-session-preference | 30 | 21/30 (70.0%) | 10/30 (33.3%) | -| **Overall** | **500** | **455/500 (91.0%)** | **411/500 (82.2%)** | - -Prompted token 消耗：总 input 13,111,421 (平均 26,275/题)，总 output 313,370 (平均 628/题)。 -平均 sessions_ingested: 44.8，dreams_triggered: 0。 - diff --git a/reme/components/prompt_handler.py b/reme/components/prompt_handler.py index 5c1a3a94..8a74a652 100644 --- a/reme/components/prompt_handler.py +++ b/reme/components/prompt_handler.py @@ -116,8 +116,7 @@ class PromptHandler: flags = {k: v for k, v in kwargs.items() if isinstance(v, bool)} formats = {k: v for k, v in kwargs.items() if not isinstance(v, bool)} - if flags: - prompt = self._apply_flag_filter(prompt, flags) + prompt = self._apply_flag_filter(prompt, flags) return prompt.format(**formats).strip() if formats else prompt diff --git a/reme/config/beam.yaml b/reme/config/beam.yaml index 430bb743..22bd9fe4 100644 --- a/reme/config/beam.yaml +++ b/reme/config/beam.yaml @@ -64,7 +64,7 @@ jobs: index_update: backend: base description: "Manually trigger incremental index update for watched dirs." - watch_dirs: [daily_dir, digest_dir, dialog_dir] + watch_dirs: [daily_dir, digest_dir, session_dir/dialog] watch_suffixes: [md, jsonl] parameters: type: object diff --git a/reme/config/lme.yaml b/reme/config/lme.yaml index 1fb4f751..2c823946 100644 --- a/reme/config/lme.yaml +++ b/reme/config/lme.yaml @@ -62,7 +62,7 @@ jobs: index_update: backend: base description: "Manually trigger incremental index update for watched dirs." - watch_dirs: [daily_dir, digest_dir, dialog_dir] + watch_dirs: [daily_dir, digest_dir, session_dir/dialog] watch_suffixes: [md, jsonl] parameters: type: object diff --git a/reme/schema/application_config.py b/reme/schema/application_config.py index 00a0b286..040df5be 100644 --- a/reme/schema/application_config.py +++ b/reme/schema/application_config.py @@ -1,8 +1,9 @@ """Application configuration models.""" import os +from pathlib import PurePosixPath, PureWindowsPath -from pydantic import BaseModel, ConfigDict, Field +from pydantic import BaseModel, ConfigDict, Field, field_validator from ..enumeration import ComponentEnum @@ -35,11 +36,11 @@ class ApplicationConfig(BaseModel): workspace_dir: str = Field(default=".reme", description="Workspace root directory for runtime files") metadata_dir: str = Field(default="metadata", description="Subdirectory for ReMe persistent state") session_dir: str = Field(default="session", description="Subdirectory for persisted agent sessions") + # dialog_dir was removed; standard transcripts are always derived as ``{session_dir}/dialog``. mem_session_dir: str = Field(default="mem_session", description="Subdirectory for persisted agent sessions") resource_dir: str = Field(default="resource", description="Subdirectory for external assets") daily_dir: str = Field(default="daily", description="Subdirectory for daily memory") digest_dir: str = Field(default="digest", description="Subdirectory for digest memory") - dialog_dir: str = Field(default="session/dialog", description="Subdirectory for dialog session transcripts") enable_logo: bool = Field(default=True, description="Show ASCII logo on startup") timezone: str | None = Field(default="Asia/Shanghai", description="IANA timezone; None uses local time") language: str = Field(default="", description="Default language for LLM interactions") @@ -53,3 +54,12 @@ class ApplicationConfig(BaseModel): default_factory=dict, description="Component registry keyed by type then name", ) + + @field_validator("session_dir") + @classmethod + def validate_session_dir(cls, value: str) -> str: + """Keep standard session storage inside the configured workspace.""" + path = value.strip() + if PurePosixPath(path.replace("\\", "/")).is_absolute() or PureWindowsPath(path).anchor: + raise ValueError("session_dir must be a workspace-relative path") + return value diff --git a/reme/steps/benchmark/lme/agentic_answer.yaml b/reme/steps/benchmark/lme/agentic_answer.yaml index 9d125086..153d211b 100644 --- a/reme/steps/benchmark/lme/agentic_answer.yaml +++ b/reme/steps/benchmark/lme/agentic_answer.yaml @@ -1,25 +1,16 @@ system_prompt: | - You are a memory retrieval assistant. A memory system manages the user's conversation history. Given a query, you MUST call the `search` tool at least once to retrieve relevant content from the memory system before answering. - - ## Memory System - The memory system stores the user's history in three forms: - - **Summary notes**: condensed knowledge distilled from past sessions. - - **Compressed session chunks**: fragments of session transcripts compressed to drop low-information phrasing; the dialogue structure is generally kept, but some detail may be shortened or lost. - - **Original sessions**: complete verbatim transcripts of past conversations, stored as memory files. - - The `search` tool retrieves summary notes and compressed session chunks; it does NOT return original sessions directly. Every result is a chunk — a fragment of a memory file identified by its path — and may lack surrounding context. Compressed session chunks are marked with a leading `compressed session chunk:` notice. When a chunk looks highly relevant to the query but its content is vague, truncated, or missing detail due to compression, use the `read` tool with the chunk's path to read the original session before drawing conclusions. - - ## Workflow - 1. Use `search` to locate possibly relevant content. It is a hybrid retrieval tool combining BM25 keyword matching and vector semantic search, ensuring diverse recall. Call it multiple times with different query phrasings to gather comprehensive information; do not stop after the first relevant result unless you are confident the information is sufficient. - 2. When a retrieved chunk is strongly related to the query but ambiguous because of compression, call `read` on its path to recover the full original session, and ground your answer on that. - 3. Use `add_draft` to save key findings during the search, and `read_all_draft` to review all saved notes before answering. - 4. After finding a preliminary answer, continue searching from multiple angles to check for potentially conflicting, supplementary, or updated information. The goal is to improve the accuracy and completeness of the final answer. - 5. Stop searching and generate the final answer once you are confident that further searches will not yield additional useful information. - - Your total tool calls should be at most 9 times. + You are a memory retrieval assistant. You MUST use the search tool to find information before answering. + - Your total time of tool calls should be at most 9 times + ## Search Strategy + - You can call 'search' tool to search multiple times (at least once) with different queries to gather comprehensive information. + - Do not stop after the first relevant one unless you are confident that the information is sufficient. + - You can use 'read' tool to read specific parts of a specific file. + - After finding a preliminary answer, continue searching from multiple angles to check for potentially conflicting, supplementary, or updated information. + ## Draft Tool + - Use 'add_draft' to save key findings during search, and 'read_all_draft' to review all saved notes before answering. ## Answer Rules - - Answer based ONLY on retrieved or read context. - - If memories contain contradictory information, weigh the context, query, and most importantly the timestamp of each memory to determine the final answer. + - Answer based ONLY on retrieved context. - Output ONLY the direct factual answer — no reasoning, no search process, no elaboration. - Do NOT invent or assume information that isn't in the memories. Do NOT extrapolate or fabricate connections from related but non-direct evidence. - If information is not found or not sufficient after multiple searches, or if the maximum iteration count is reached without finding answer, reply: 'Information not found.' diff --git a/reme/steps/evolve/auto_memory.py b/reme/steps/evolve/auto_memory.py index 0277fd5f..1c2aaa44 100644 --- a/reme/steps/evolve/auto_memory.py +++ b/reme/steps/evolve/auto_memory.py @@ -10,6 +10,7 @@ from ._evolve import agent_reply_result_text, format_history, now from ..base_step import BaseStep from ..file_io import extract_daily_date, parse_daily_date, refresh_day_index from ..file_io import validate_filename_component, validate_session_id +from ..index import normalize_posix_path from ...components import R _SESSION_ID_KEY = "session_id" @@ -66,13 +67,13 @@ class AutoMemoryStep(BaseStep): self.update_tools: list[str] = ["read", "edit", "frontmatter_update", "write"] def _session_dir(self) -> str: - return str(self.config_value("session_dir")).strip("/") + return normalize_posix_path(str(self.config_value("session_dir"))) def _session_path(self, session_id: str) -> Path: return self.file_store.workspace_path / self._session_dir() / "dialog" / f"{session_id}.jsonl" def _session_source_path(self, session_id: str) -> str: - return f"{self._session_dir()}/dialog/{session_id}.jsonl" + return normalize_posix_path(f"{self._session_dir()}/dialog/{session_id}.jsonl") def _session_link(self, session_id: str) -> str: return f"[[{self._session_source_path(session_id)}]]" diff --git a/reme/steps/evolve/auto_memory_cc.py b/reme/steps/evolve/auto_memory_cc.py index 43b23219..b1e7c256 100644 --- a/reme/steps/evolve/auto_memory_cc.py +++ b/reme/steps/evolve/auto_memory_cc.py @@ -27,6 +27,7 @@ from pathlib import Path from typing import Any from .auto_memory import AutoMemoryStep +from ..index import normalize_posix_path from ...components import R from ...components.agent_wrapper import CcFileSessionStore @@ -73,7 +74,8 @@ class AutoMemoryCCStep(AutoMemoryStep): return def _session_link(self, session_id: str) -> str: - return f"[[{self._session_dir()}/{self._CC_STORE_SUBDIR}/{session_id}.jsonl]]" + path = normalize_posix_path(f"{self._session_dir()}/{self._CC_STORE_SUBDIR}/{session_id}.jsonl") + return f"[[{path}]]" # ----- session: Claude Code side <-> ReMe CC SessionStore ---------------- diff --git a/reme/steps/evolve/dream/integrate.yaml b/reme/steps/evolve/dream/integrate.yaml index 6b52cb46..25d66250 100644 --- a/reme/steps/evolve/dream/integrate.yaml +++ b/reme/steps/evolve/dream/integrate.yaml @@ -18,8 +18,8 @@ integrate_system_prompt_procedure: | genuinely needs more. - If the draft starts copying paragraphs or narrating the event, it belongs in the source material, not digest. - - The Sources section carries details: cite every relevant unit_paths entry - as `- [[]]`. + - The Sources section carries provenance: cite every relevant unit_paths + entry inside a complete sentence that explains what the material supports. - Digest-to-digest wikilinks carry the conceptual graph. ## Procedure Body Shape @@ -30,9 +30,10 @@ integrate_system_prompt_procedure: | - Steps: numbered or terse bullets; each step is verb-led. - Pre-conditions / inputs: short list, not prose. - Failure modes / caveats: brief. - - `## Sources`: at least one `- [[]]` item, and normally one - for every relevant path in unit_paths. Plain-prose provenance does NOT - count; only wikilinks survive future updates. + - `## Sources`: at least one sentence containing `[[]]`, and + normally one for every relevant path in unit_paths. A bare wikilink does + not count; every link must have surrounding prose that explains what the + source supports. ## Workflow @@ -60,12 +61,18 @@ integrate_system_prompt_procedure: | UPDATE. UPDATE must be additive: never remove existing wikilinks or source links. Default to weaving more, not less; this is the only chance to attach recalled related nodes. + 5. Before returning, read the target and rewrite any standalone relation + field or bare-link line into a contextual sentence. ## Wikilink Graph - - A `## Sources` section points from digest back to material with bare list - items such as `- [[daily//.md]]` and - `- [[resource/]]`. + - A `## Sources` section points from digest back to material. Each entry must + be a complete contextual sentence, for example `- This workflow was + observed in [[daily//.md|the session record]].` + - Never emit standalone relation fields or bare-link lines such as + `relates_to:: [[...]]`, `derived_from:: [[...]]`, or `- [[...]]`. + Digest links belong inside the sentence that explains the relationship; + source links belong inside the sentence that explains the evidence. - Procedure nodes may link to any digest bucket: `[[{digest_dir}/procedure/.md]]`, `[[{digest_dir}/personal/.md]]`, or @@ -98,7 +105,7 @@ integrate_system_prompt_procedure_zh: | - 正文短且抽象，通常 50-200 words；只有流程本身确实需要时才更长。 - 如果草稿开始复制段落或叙述事件，说明细节放错层了。 - - Sources 章节承载细节：用 `- [[]]` 引用 unit_paths 中每个相关来源。 + - Sources 章节承载 provenance：在完整句子中引用 unit_paths 中每个相关来源，并说明材料支持什么。 - Digest 之间的 wikilink 承载概念图。 ## Procedure 正文形态 @@ -109,8 +116,8 @@ integrate_system_prompt_procedure_zh: | - Steps：编号或短 bullet；每步以动词开头。 - Pre-conditions / inputs：短列表，不写长 prose。 - Failure modes / caveats：简短。 - - `## Sources`：至少一条 `- [[]]`，通常覆盖 unit_paths 中每个相关 path。 - 纯文本 provenance 不算；只有 wikilink 会在未来更新中保留下来。 + - `## Sources`：至少一个包含 `[[]]` 的完整句子，通常覆盖 unit_paths 中每个相关 path。 + 裸 Wikilink 不算；每个链接周围都必须有自然语言，说明该来源支持什么。 ## 工作流 @@ -129,11 +136,14 @@ integrate_system_prompt_procedure_zh: | `> note: contradicted by [[]] - ` 内联标注。 4. CREATE 和 UPDATE 都要把 related digest 节点织入正文 wikilink。UPDATE 必须只增不删：不要删除已有 wikilink 或来源链接。默认多织，而不是少织；这是挂接召回到的相关节点的唯一机会。 + 5. 返回前读取目标文件，把任何独立关系字段或裸链接行改写成有上下文的自然句子。 ## Wikilink 图 - - `## Sources` 章节用普通列表 Wikilink 从 digest 指回材料，例如 - `- [[daily//.md]]` 和 `- [[resource/]]`。 + - `## Sources` 章节从 digest 指回材料。每一项必须是有上下文的完整句子，例如 + `- 这一流程见于 [[daily//.md|该会话记录]]。` + - 禁止生成 `relates_to:: [[...]]`、`derived_from:: [[...]]` 或 `- [[...]]` 这类独立关系字段和裸链接行。 + Digest 链接必须融入解释关系的句子；来源链接必须融入解释证据的句子。 - Procedure 节点可以链接任意 digest bucket： `[[{digest_dir}/procedure/.md]]`、`[[{digest_dir}/personal/.md]]`、 `[[{digest_dir}/wiki/.md]]`。 @@ -166,8 +176,8 @@ integrate_system_prompt_personal: | - Body is short and operational, usually 50-200 words. - Do not narrate what the user said in detail; cite source material instead. - - The Sources section carries details: cite every relevant unit_paths entry - as `- [[]]`. + - The Sources section carries provenance: cite every relevant unit_paths + entry inside a complete sentence that explains what the material supports. - Digest-to-digest wikilinks carry the conceptual graph. ## Personal Body Shape @@ -180,9 +190,10 @@ integrate_system_prompt_personal: | - `How to apply:` contexts, tasks, boundaries, or exceptions. - Do not invent exceptions or soften a hard preference unless the source material explicitly supports that exception. - - `## Sources`: at least one `- [[]]` item, and normally one - for every relevant path in unit_paths. Plain-prose provenance does NOT - count; only wikilinks survive future updates. + - `## Sources`: at least one sentence containing `[[]]`, and + normally one for every relevant path in unit_paths. A bare wikilink does + not count; every link must have surrounding prose that explains what the + source supports. For preferences, prefer one node per preference rather than one large person node; that is the granularity downstream search will hit. @@ -213,12 +224,18 @@ integrate_system_prompt_personal: | UPDATE. UPDATE must be additive: never remove existing wikilinks or source links. Default to weaving more, not less; this is the only chance to attach recalled related nodes. + 5. Before returning, read the target and rewrite any standalone relation + field or bare-link line into a contextual sentence. ## Wikilink Graph - - A `## Sources` section points from digest back to material with bare list - items such as `- [[daily//.md]]` and - `- [[resource/]]`. + - A `## Sources` section points from digest back to material. Each entry must + be a complete contextual sentence, for example `- This preference was + observed in [[daily//.md|the session record]].` + - Never emit standalone relation fields or bare-link lines such as + `relates_to:: [[...]]`, `derived_from:: [[...]]`, or `- [[...]]`. + Digest links belong inside the sentence that explains the relationship; + source links belong inside the sentence that explains the evidence. - Personal nodes may link to any digest bucket: `[[{digest_dir}/personal/.md]]`, `[[{digest_dir}/procedure/.md]]`, or @@ -252,7 +269,7 @@ integrate_system_prompt_personal_zh: | - 正文短且可操作，通常 50-200 words。 - 不要详细复述用户说了什么；用来源材料承载细节。 - - Sources 章节承载细节：用 `- [[]]` 引用 unit_paths 中每个相关来源。 + - Sources 章节承载 provenance：在完整句子中引用 unit_paths 中每个相关来源，并说明材料支持什么。 - Digest 之间的 wikilink 承载概念图。 ## Personal 正文形态 @@ -263,8 +280,8 @@ integrate_system_prompt_personal_zh: | - `Why:` 原因或上下文，帮助未来判断边界情况。 - `How to apply:` 适用上下文、任务、边界或例外。 - 不要凭空添加例外，也不要软化明确偏好；只有来源材料明确支持时才写例外。 - - `## Sources`：至少一条 `- [[]]`，通常覆盖 unit_paths 中每个相关 path。 - 纯文本 provenance 不算；只有 wikilink 会在未来更新中保留下来。 + - `## Sources`：至少一个包含 `[[]]` 的完整句子，通常覆盖 unit_paths 中每个相关 path。 + 裸 Wikilink 不算；每个链接周围都必须有自然语言，说明该来源支持什么。 偏好类内容优先一条偏好一个 node，而不是一个人一个大 node；这是下游搜索更容易命中的粒度。 @@ -285,11 +302,14 @@ integrate_system_prompt_personal_zh: | `> note: contradicted by [[]] - ` 内联标注。 4. CREATE 和 UPDATE 都要把 related digest 节点织入正文 wikilink。UPDATE 必须只增不删：不要删除已有 wikilink 或来源链接。默认多织，而不是少织；这是挂接召回到的相关节点的唯一机会。 + 5. 返回前读取目标文件，把任何独立关系字段或裸链接行改写成有上下文的自然句子。 ## Wikilink 图 - - `## Sources` 章节用普通列表 Wikilink 从 digest 指回材料，例如 - `- [[daily//.md]]` 和 `- [[resource/]]`。 + - `## Sources` 章节从 digest 指回材料。每一项必须是有上下文的完整句子，例如 + `- 这一偏好见于 [[daily//.md|该会话记录]]。` + - 禁止生成 `relates_to:: [[...]]`、`derived_from:: [[...]]` 或 `- [[...]]` 这类独立关系字段和裸链接行。 + Digest 链接必须融入解释关系的句子；来源链接必须融入解释证据的句子。 - Personal 节点可以链接任意 digest bucket： `[[{digest_dir}/personal/.md]]`、`[[{digest_dir}/procedure/.md]]`、 `[[{digest_dir}/wiki/.md]]`。 @@ -325,8 +345,8 @@ integrate_system_prompt_wiki: | genuinely needs more. - If the draft starts copying paragraphs or narrating the event, it belongs in the source material, not digest. - - The Sources section carries details: cite every relevant unit_paths entry - as `- [[]]`. + - The Sources section carries provenance: cite every relevant unit_paths + entry inside a complete sentence that explains what the material supports. - Digest-to-digest wikilinks carry the conceptual graph. ## Wiki Body Shape @@ -337,9 +357,10 @@ integrate_system_prompt_wiki: | - Body: short paragraphs or tight bullets with properties, sub-claims, distinctions, and one-line examples. - Relations: weave related nodes into natural prose with ordinary wikilinks. - - `## Sources`: at least one `- [[]]` item, and normally one - for every relevant path in unit_paths. Plain-prose provenance does NOT - count; only wikilinks survive future updates. + - `## Sources`: at least one sentence containing `[[]]`, and + normally one for every relevant path in unit_paths. A bare wikilink does + not count; every link must have surrounding prose that explains what the + source supports. ## Workflow @@ -368,12 +389,18 @@ integrate_system_prompt_wiki: | UPDATE. UPDATE must be additive: never remove existing wikilinks or source links. Default to weaving more, not less; this is the only chance to attach recalled related nodes. + 5. Before returning, read the target and rewrite any standalone relation + field or bare-link line into a contextual sentence. ## Wikilink Graph - - A `## Sources` section points from digest back to material with bare list - items such as `- [[daily//.md]]` and - `- [[resource/]]`. + - A `## Sources` section points from digest back to material. Each entry must + be a complete contextual sentence, for example `- This claim is supported + by [[daily//.md|the session record]].` + - Never emit standalone relation fields or bare-link lines such as + `relates_to:: [[...]]`, `derived_from:: [[...]]`, or `- [[...]]`. + Digest links belong inside the sentence that explains the relationship; + source links belong inside the sentence that explains the evidence. - Wiki nodes may link to any digest bucket: `[[{digest_dir}/wiki/.md]]`, `[[{digest_dir}/procedure/.md]]`, or @@ -407,7 +434,7 @@ integrate_system_prompt_wiki_zh: | - 正文短且抽象，通常 50-200 words；只有概念本身确实需要时才更长。 - 如果草稿开始复制段落或叙述事件，说明细节放错层了。 - - Sources 章节承载细节：用 `- [[]]` 引用 unit_paths 中每个相关来源。 + - Sources 章节承载 provenance：在完整句子中引用 unit_paths 中每个相关来源，并说明材料支持什么。 - Digest 之间的 wikilink 承载概念图。 ## Wiki 正文形态 @@ -417,8 +444,8 @@ integrate_system_prompt_wiki_zh: | - First line：一句话定义或主张。 - Body：短段落或紧凑 bullets，写属性、子主张、区分和一行例子。 - Relations：在自然语言中用普通 Wikilink 织入相关节点。 - - `## Sources`：至少一条 `- [[]]`，通常覆盖 unit_paths 中每个相关 path。 - 纯文本 provenance 不算；只有 wikilink 会在未来更新中保留下来。 + - `## Sources`：至少一个包含 `[[]]` 的完整句子，通常覆盖 unit_paths 中每个相关 path。 + 裸 Wikilink 不算；每个链接周围都必须有自然语言，说明该来源支持什么。 ## 工作流 @@ -437,11 +464,14 @@ integrate_system_prompt_wiki_zh: | `> note: contradicted by [[]] - ` 内联标注。 4. CREATE 和 UPDATE 都要把 related digest 节点织入正文 wikilink。UPDATE 必须只增不删：不要删除已有 wikilink 或来源链接。默认多织，而不是少织；这是挂接召回到的相关节点的唯一机会。 + 5. 返回前读取目标文件，把任何独立关系字段或裸链接行改写成有上下文的自然句子。 ## Wikilink 图 - - `## Sources` 章节用普通列表 Wikilink 从 digest 指回材料，例如 - `- [[daily//.md]]` 和 `- [[resource/]]`。 + - `## Sources` 章节从 digest 指回材料。每一项必须是有上下文的完整句子，例如 + `- 这一主张由 [[daily//.md|该会话记录]] 支持。` + - 禁止生成 `relates_to:: [[...]]`、`derived_from:: [[...]]` 或 `- [[...]]` 这类独立关系字段和裸链接行。 + Digest 链接必须融入解释关系的句子；来源链接必须融入解释证据的句子。 - Wiki 节点可以链接任意 digest bucket： `[[{digest_dir}/wiki/.md]]`、`[[{digest_dir}/procedure/.md]]`、 `[[{digest_dir}/personal/.md]]`。 @@ -471,8 +501,9 @@ integrate_user_message: | {material_blob} Integrate this single unit into digest memory. Cite every relevant unit_paths - entry as `- [[]]` under `## Sources`, recall related digest nodes, and - weave useful digest wikilinks into the target node. + entry in a contextual sentence under `## Sources`, recall related digest + nodes, and weave useful digest wikilinks into natural body prose. Never emit + standalone relation fields or bare-link lines. integrate_user_message_zh: | 提示：{hint} @@ -488,5 +519,6 @@ integrate_user_message_zh: | {material_blob} - 将这个单独的 unit 整合进 digest memory。在 `## Sources` 下用 `- [[]]` 引用 unit_paths 中每个 - 相关来源，召回相关 digest 节点，并把有用的 digest wikilink 织入目标节点。 + 将这个单独的 unit 整合进 digest memory。在 `## Sources` 下用有上下文的完整句子引用 unit_paths 中每个 + 相关来源，召回相关 digest 节点，并把有用的 digest Wikilink 融入正文的自然表达。禁止生成独立关系字段 + 或裸链接行。 diff --git a/reme/steps/file_io/daily_write.py b/reme/steps/file_io/daily_write.py index cf0f4f6a..31ec050b 100644 --- a/reme/steps/file_io/daily_write.py +++ b/reme/steps/file_io/daily_write.py @@ -3,6 +3,7 @@ from ._daily_index import parse_daily_date, refresh_day_index, validate_session_id from ._path import validate_filename_component from ..base_step import BaseStep +from ..index import normalize_posix_path from ...components import R from ...steps.evolve import now @@ -21,10 +22,11 @@ class DailyWriteStep(BaseStep): self.context.response.metadata.update(meta) def _session_dir(self) -> str: - return str(self.config_value("session_dir")).strip("/") + return normalize_posix_path(str(self.config_value("session_dir"))) def _session_link(self, session_id: str) -> str: - return f"[[{self._session_dir()}/dialog/{session_id}.jsonl]]" + path = normalize_posix_path(f"{self._session_dir()}/dialog/{session_id}.jsonl") + return f"[[{path}]]" def _collect_required(self) -> tuple[str, str, str, str] | None: assert self.context is not None diff --git a/reme/steps/file_io/read.py b/reme/steps/file_io/read.py index 0df2342a..48f53aea 100644 --- a/reme/steps/file_io/read.py +++ b/reme/steps/file_io/read.py @@ -109,13 +109,13 @@ class ReadStep(BaseStep): the workspace or is not a raw session transcript. """ try: - rel_path = str(target.relative_to(self.workspace_path)) + rel_path = target.relative_to(self.workspace_path).as_posix() except ValueError: self.logger.info(f"[{self.name}] skip session format: path outside workspace_path path={target}") return truncate_text_output(excerpt, start_line=start_line, total_lines=total, file_path=str(target)) - dialog_dir = self.config_value("dialog_dir") - if not is_session_path(rel_path, dialog_dir): + session_dir = self.config_value("session_dir") + if not is_session_path(rel_path, session_dir): return truncate_text_output(excerpt, start_line=start_line, total_lines=total, file_path=str(target)) return truncate_session_output( diff --git a/reme/steps/index/__init__.py b/reme/steps/index/__init__.py index 51727e27..4c66c608 100644 --- a/reme/steps/index/__init__.py +++ b/reme/steps/index/__init__.py @@ -1,5 +1,6 @@ """Index steps.""" +from ._source_format import normalize_posix_path from .bm25_search import Bm25SearchStep from .clear_paths import ClearPathsStep from .clear_store import ClearStoreStep @@ -35,6 +36,7 @@ __all__ = [ "InitChangesStep", "LogChangesStep", "NodeSearchStep", + "normalize_posix_path", "ReadAllDraftStep", "OptimizeIndexStep", "SearchStep", diff --git a/reme/steps/index/_source_format.py b/reme/steps/index/_source_format.py index d7a46db9..34bb182d 100644 --- a/reme/steps/index/_source_format.py +++ b/reme/steps/index/_source_format.py @@ -20,6 +20,7 @@ removes every previously-returned result. :data:`NO_RESULTS_MESSAGE` is the English notice shown when the search returned no results at all. """ +import posixpath from typing import Callable, Final from agentscope.message import Msg @@ -38,21 +39,28 @@ ALL_RETURNED_MESSAGE: Final[str] = ( NO_RESULTS_MESSAGE: Final[str] = "No relevant information was found for the given query." -def is_session_path(path: str, dialog_dir: str) -> bool: - """True if the path points at a raw session transcript (a jsonl file under the dialog dir).""" - path = (path or "").strip().strip("/") +def normalize_posix_path(path: str) -> str: + """Return a normalized, workspace-relative POSIX path string.""" + normalized = posixpath.normpath((path or "").strip().replace("\\", "/").strip("/")) + return "" if normalized == "." else normalized + + +def is_session_path(path: str, session_dir: str) -> bool: + """True if the path points at a raw transcript under ``{session_dir}/dialog``.""" + path = normalize_posix_path(path) if not path.endswith(".jsonl"): return False - dialog_dir = (dialog_dir or "").strip("/") - return path == dialog_dir or path.startswith(f"{dialog_dir}/") + session_root = normalize_posix_path(session_dir) + session_dialog_dir = posixpath.join(session_root, "dialog") + return path == session_dialog_dir or path.startswith(f"{session_dialog_dir}/") -def is_session_chunk(chunk: FileChunk, dialog_dir: str) -> bool: +def is_session_chunk(chunk: FileChunk, session_dir: str) -> bool: """True if the chunk comes from a raw session transcript (a jsonl file under the dialog dir).""" - return is_session_path(chunk.path, dialog_dir) + return is_session_path(chunk.path, session_dir) -def render_chunk_body(chunk: FileChunk, dialog_dir: str) -> str: +def render_chunk_body(chunk: FileChunk, session_dir: str) -> str: """Render a chunk's body; raw session transcripts render one message per line. Session chunks are jsonl where each line is a serialized ``Msg``. They @@ -61,7 +69,7 @@ def render_chunk_body(chunk: FileChunk, dialog_dir: str) -> str: share the same single-line-per-message format. All other chunks keep their raw ``text``. """ - if not is_session_chunk(chunk, dialog_dir): + if not is_session_chunk(chunk, session_dir): return chunk.text return "\n".join(render_session_chunk_lines(chunk)) @@ -123,7 +131,7 @@ def _build_union_chunk(group: list[FileChunk]) -> FileChunk: ) -def merge_session_chunk_intervals(chunks: list[FileChunk], dialog_dir: str) -> list[FileChunk]: +def merge_session_chunk_intervals(chunks: list[FileChunk], session_dir: str) -> list[FileChunk]: """Merge raw session chunks from the same file into their line-range union. Only chunks recognized as raw session transcripts (see @@ -148,7 +156,7 @@ def merge_session_chunk_intervals(chunks: list[FileChunk], dialog_dir: str) -> l # non-session chunks use their own rank and thus keep their position. ordered: list[tuple[int, int, FileChunk]] = [] for idx, c in enumerate(chunks): - if is_session_chunk(c, dialog_dir): + if is_session_chunk(c, session_dir): session_by_path.setdefault(c.path, []).append((idx, c)) else: ordered.append((idx, c.start_line, c)) @@ -183,7 +191,7 @@ def _finalize_group(group: list[FileChunk]) -> FileChunk: def render_chunk_entries( chunks: list[FileChunk], - dialog_dir: str, + session_dir: str, *, include_source: bool = True, score_fn: Callable[[FileChunk], str] | None = None, @@ -207,7 +215,7 @@ def render_chunk_entries( expansion lines (used by hybrid search). """ if not include_source: - return [{"body": render_chunk_body(c, dialog_dir)} for c in chunks] + return [{"body": render_chunk_body(c, session_dir)} for c in chunks] fmt = score_fn or (lambda c: f"score={c.score:.4f}") entries: list[dict[str, str]] = [] @@ -218,7 +226,7 @@ def render_chunk_entries( "start_line": str(c.start_line), "end_line": str(c.end_line), "score": fmt(c), - "body": render_chunk_body(c, dialog_dir), + "body": render_chunk_body(c, session_dir), "link": "\n".join(render_expansion_lines((link_expansion or {}).get(c.path, {}))), }, ) diff --git a/reme/steps/index/_watch_rules.py b/reme/steps/index/_watch_rules.py index 739d6138..352c2f92 100644 --- a/reme/steps/index/_watch_rules.py +++ b/reme/steps/index/_watch_rules.py @@ -4,6 +4,8 @@ from dataclasses import dataclass, field from pathlib import Path from typing import TYPE_CHECKING +from ._source_format import normalize_posix_path + if TYPE_CHECKING: from ...schema import ApplicationConfig from ...components.runtime_context import RuntimeContext @@ -27,8 +29,19 @@ def build_watch_rules( """Build watch rules from application config fields and suffix whitelist.""" rules: list[WatchRule] = [] for dir_field in watch_dirs: - dir_name = getattr(app_config, dir_field, dir_field) - rules.append(WatchRule(path=workspace_path / dir_name, suffixes=list(watch_suffixes))) + literal_path = Path(dir_field) + if literal_path.is_absolute(): + rule_path = literal_path + else: + config_field, separator, child_path = dir_field.partition("/") + dir_value = getattr(app_config, config_field, config_field) + if hasattr(app_config, config_field) and dir_value in (None, ""): + dir_value = getattr(type(app_config)(), config_field) + if separator: + dir_value = normalize_posix_path(f"{dir_value}/{child_path}") + dir_name = Path(dir_value) + rule_path = dir_name if dir_name.is_absolute() else workspace_path / dir_name + rules.append(WatchRule(path=rule_path, suffixes=list(watch_suffixes))) return rules diff --git a/reme/steps/index/bm25_search.py b/reme/steps/index/bm25_search.py index f2fc4998..3ac139c3 100644 --- a/reme/steps/index/bm25_search.py +++ b/reme/steps/index/bm25_search.py @@ -48,10 +48,10 @@ class Bm25SearchStep(_ToolContextDedupMixin, BaseStep): else: results = results[:limit] - dialog_dir = self.config_value("dialog_dir") + session_dir = self.config_value("session_dir") entries = render_chunk_entries( - merge_session_chunk_intervals(results, dialog_dir), - dialog_dir, + merge_session_chunk_intervals(results, session_dir), + session_dir, include_source=self.include_source, ) self.context.response.answer = join_chunk_entries(entries) diff --git a/reme/steps/index/search_v2.py b/reme/steps/index/search_v2.py index bdeb91e0..558eefc4 100644 --- a/reme/steps/index/search_v2.py +++ b/reme/steps/index/search_v2.py @@ -213,15 +213,15 @@ class SearchV2Step(_ToolContextDedupMixin, BaseStep): await expand_links(self.file_store, unique_paths, max_links_per_direction) if expand_links_enabled else {} ) - dialog_dir = self.config_value("dialog_dir") + session_dir = self.config_value("session_dir") entries = render_chunk_entries( - merge_session_chunk_intervals(fused, dialog_dir), - dialog_dir, + merge_session_chunk_intervals(fused, session_dir), + session_dir, score_fn=lambda c: self._format_scores(c.scores, hybrid), link_expansion=link_expansion, ) if self._session_compress_enabled(): - await self._compress_session_entries(entries, query, dialog_dir) + await self._compress_session_entries(entries, query, session_dir) self.context.response.answer = join_chunk_entries(entries) if not fused: self.context.response.answer = ALL_RETURNED_MESSAGE if pre_dedup_count > 0 else NO_RESULTS_MESSAGE @@ -246,7 +246,7 @@ class SearchV2Step(_ToolContextDedupMixin, BaseStep): value = (search_cfg.get("_compress") or {}).get("session") return value is True or str(value).strip().lower() == "true" - async def _compress_session_entries(self, entries: list[dict[str, str]], query: str, dialog_dir: str) -> None: + async def _compress_session_entries(self, entries: list[dict[str, str]], query: str, session_dir: str) -> None: """Compress session-transcript entry bodies in place via the ``compressor`` job. Only entries whose ``path`` points at a raw session transcript are @@ -323,7 +323,7 @@ class SearchV2Step(_ToolContextDedupMixin, BaseStep): return entry["body"] = f"compressed session chunk:\n{compressed}" - targets = [e for e in entries if is_session_path(e.get("path", ""), dialog_dir)] + targets = [e for e in entries if is_session_path(e.get("path", ""), session_dir)] if not targets: return self.logger.info(f"[{self.name}] compressing {len(targets)} session entries with {len(queries)} queries") diff --git a/reme/steps/index/vector_search.py b/reme/steps/index/vector_search.py index c1c11f42..638344e1 100644 --- a/reme/steps/index/vector_search.py +++ b/reme/steps/index/vector_search.py @@ -48,10 +48,10 @@ class VectorSearchStep(_ToolContextDedupMixin, BaseStep): else: results = results[:limit] - dialog_dir = self.config_value("dialog_dir") + session_dir = self.config_value("session_dir") entries = render_chunk_entries( - merge_session_chunk_intervals(results, dialog_dir), - dialog_dir, + merge_session_chunk_intervals(results, session_dir), + session_dir, include_source=self.include_source, ) self.context.response.answer = join_chunk_entries(entries) diff --git a/tests/unit/test_application_config.py b/tests/unit/test_application_config.py new file mode 100644 index 00000000..df35bf89 --- /dev/null +++ b/tests/unit/test_application_config.py @@ -0,0 +1,148 @@ +"""Application path configuration contracts.""" + +# pylint: disable=protected-access + +from types import SimpleNamespace + +import pytest +from agentscope.message import Msg +from pydantic import ValidationError + +from reme.application import Application +from reme.components import ApplicationContext +from reme.schema import ApplicationConfig, FileChunk +from reme.steps.evolve.auto_memory import AutoMemoryStep +from reme.steps.file_io.daily_write import DailyWriteStep +from reme.steps.index._source_format import is_session_path, render_chunk_body +from reme.steps.index._watch_rules import build_watch_rules + + +def test_dialog_dir_is_not_an_application_config_field(): + """The removed option is absent from schemas and ignored when supplied.""" + custom = ApplicationConfig(session_dir="sessions/", dialog_dir="somewhere/else") + + assert not hasattr(custom, "dialog_dir") + assert "dialog_dir" not in custom.model_dump() + assert "dialog_dir" not in ApplicationConfig.model_json_schema()["properties"] + + +@pytest.mark.parametrize("session_dir", ["/tmp/sessions", "C:\\sessions", "\\\\server\\share\\sessions"]) +def test_application_rejects_absolute_session_dir_before_workspace_setup(tmp_path, session_dir): + """Application initialization rejects session directories outside the workspace.""" + workspace = tmp_path / "workspace" + + with pytest.raises(ValidationError, match="session_dir must be a workspace-relative path"): + Application(workspace_dir=str(workspace), session_dir=session_dir) + + assert not workspace.exists() + + +def test_dialog_watch_rule_follows_custom_session_dir(tmp_path): + """Nested watch rules resolve the dialog directory below session_dir.""" + config = ApplicationConfig(session_dir="sessions") + + rules = build_watch_rules( + config, + tmp_path, + watch_dirs=["session_dir/dialog"], + watch_suffixes=["jsonl"], + ) + + assert len(rules) == 1 + assert rules[0].path == tmp_path / "sessions" / "dialog" + assert rules[0].suffixes == ["jsonl"] + + +def test_dialog_watch_rule_matches_writer_for_normalized_session_dirs(tmp_path): + """Nested watch rules use the same defaults and normalization as transcript writers.""" + for configured in ("", "parent//../sessions"): + app_context = ApplicationContext(session_dir=configured) + auto_memory = AutoMemoryStep(app_context=app_context) + auto_memory.file_store = SimpleNamespace(workspace_path=tmp_path) + + rules = build_watch_rules( + app_context.app_config, + tmp_path, + watch_dirs=["session_dir/dialog"], + watch_suffixes=["jsonl"], + ) + + assert rules[0].path == auto_memory._session_path("s1").parent + + +def test_absolute_watch_rule_stays_outside_workspace(tmp_path): + """Absolute literal watch directories retain their original location.""" + config = ApplicationConfig() + absolute_dir = tmp_path.parent / "reme-dialogs" + + rules = build_watch_rules( + config, + tmp_path, + watch_dirs=[str(absolute_dir)], + watch_suffixes=["jsonl"], + ) + + assert rules[0].path == absolute_dir + + +def test_standard_session_paths_and_links_follow_custom_session_dir(tmp_path): + """Writers derive transcript paths and links from session_dir.""" + app_context = ApplicationContext(session_dir="sessions") + auto_memory = AutoMemoryStep(app_context=app_context) + auto_memory.file_store = SimpleNamespace(workspace_path=tmp_path) + daily_write = DailyWriteStep(app_context=app_context) + + assert auto_memory._session_path("s1") == tmp_path / "sessions" / "dialog" / "s1.jsonl" + assert auto_memory._session_source_path("s1") == "sessions/dialog/s1.jsonl" + assert auto_memory._session_link("s1") == "[[sessions/dialog/s1.jsonl]]" + assert daily_write._session_link("s1") == "[[sessions/dialog/s1.jsonl]]" + + +def test_session_paths_and_links_normalize_custom_session_dir(tmp_path): + """Writers and source links use the same normalized POSIX session root.""" + for configured, expected in ( + ("./sessions", "sessions"), + (".", ""), + ("parent//../sessions", "sessions"), + ): + app_context = ApplicationContext(session_dir=configured) + auto_memory = AutoMemoryStep(app_context=app_context) + auto_memory.file_store = SimpleNamespace(workspace_path=tmp_path) + daily_write = DailyWriteStep(app_context=app_context) + source_path = f"{expected + '/' if expected else ''}dialog/s1.jsonl" + + assert auto_memory._session_path("s1") == tmp_path / source_path + assert auto_memory._session_source_path("s1") == source_path + assert auto_memory._session_link("s1") == f"[[{source_path}]]" + assert daily_write._session_link("s1") == f"[[{source_path}]]" + + +def test_dialog_classification_excludes_other_session_stores(): + """Only jsonl files below the dialog child directory are transcripts.""" + assert is_session_path("sessions/dialog/s1.jsonl", "sessions") + assert not is_session_path("sessions/claude_code/s1.jsonl", "sessions") + assert not is_session_path("session/dialog/s1.jsonl", "sessions") + + +def test_dialog_classification_normalizes_paths(): + """Session recognition compares normalized path representations.""" + assert is_session_path("sessions/dialog/s1.jsonl", "./sessions") + assert is_session_path("./sessions//dialog/s1.jsonl", "sessions") + assert is_session_path("dialog/s1.jsonl", ".") + assert is_session_path("sessions/dialog/s1.jsonl", "parent/../sessions") + + +def test_session_rendering_follows_custom_session_dir(): + """Session-aware rendering recognizes a custom session root.""" + message = Msg(name="user", role="user", content=[{"type": "text", "text": "hello"}]) + chunk = FileChunk( + path="sessions/dialog/s1.jsonl", + start_line=1, + end_line=1, + text=message.model_dump_json(), + ) + + rendered = render_chunk_body(chunk, "sessions") + + assert rendered.startswith("[user @") + assert rendered.endswith("] hello") diff --git a/tests/unit/test_auto_dream.py b/tests/unit/test_auto_dream.py index 9577d999..9509d67d 100644 --- a/tests/unit/test_auto_dream.py +++ b/tests/unit/test_auto_dream.py @@ -156,6 +156,22 @@ def test_parse_structured_reply_handles_fenced_yaml_and_scalar_fallback(): assert data["note"].startswith("Extended node") +def test_integrate_prompts_require_wikilinks_in_contextual_prose(): + """Every digest bucket rejects standalone relation fields and bare links.""" + prompt_path = Path(__file__).parents[2] / "reme" / "steps" / "evolve" / "dream" / "integrate.yaml" + prompts = yaml.safe_load(prompt_path.read_text(encoding="utf-8")) + + for bucket in ("procedure", "personal", "wiki"): + english = prompts[f"integrate_system_prompt_{bucket}"] + chinese = prompts[f"integrate_system_prompt_{bucket}_zh"] + assert "Never emit standalone relation fields or bare-link lines" in english + assert "Digest links belong inside the sentence" in english + assert "Before returning, read the target" in english + assert "禁止生成 `relates_to:: [[...]]`" in chinese + assert "Digest 链接必须融入解释关系的句子" in chinese + assert "返回前读取目标文件" in chinese + + def test_extract_clean_output_respects_max_units(): """Extract cleaning caps valid units at max_units.""" state = DreamState(changed_paths=["daily/a.md"]) diff --git a/tests/unit/test_crud_steps.py b/tests/unit/test_crud_steps.py index 1132a686..b08bd8b9 100644 --- a/tests/unit/test_crud_steps.py +++ b/tests/unit/test_crud_steps.py @@ -31,7 +31,9 @@ import os import tempfile import warnings from pathlib import Path, PureWindowsPath +from types import SimpleNamespace +from reme.components import ApplicationContext from reme.components.file_store import LocalFileStore from reme.schema import FileNode from reme.steps.file_io import ( @@ -811,6 +813,31 @@ def test_read_format_session_injected_false_disables_yaml_true(): _run(run()) +def test_read_format_session_with_non_normalized_session_dir(): + """Session formatting recognizes a normalized target when config contains ``./``.""" + + async def run(): + from agentscope.message import Msg + + m = Msg(name="user", role="user", content=[{"type": "text", "text": "hello world"}]) + jsonl_body = m.model_dump_json() + "\n" + + with tempfile.TemporaryDirectory() as tmp: + workspace = Path(tmp) + session_path = workspace / "sessions" / "dialog" / "s.jsonl" + session_path.parent.mkdir(parents=True, exist_ok=True) + session_path.write_text(jsonl_body, encoding="utf-8") + + app_context = ApplicationContext(workspace_dir=str(workspace.resolve()), session_dir="./sessions") + step = crud_read.ReadStep(app_context=app_context, read_step_format_session=True) + step.file_store = SimpleNamespace(workspace_path=workspace) + await step(path="sessions/dialog/s.jsonl") + + assert "[user @" in str(step.context.response.answer) + + _run(run()) + + # -- write / edit -------------------------------------------------------- diff --git a/tests/unit/test_prompt_handler.py b/tests/unit/test_prompt_handler.py index 2678d7dc..a07cd2c6 100644 --- a/tests/unit/test_prompt_handler.py +++ b/tests/unit/test_prompt_handler.py @@ -182,13 +182,18 @@ def test_flag_filter_removes_non_matching(): def test_flag_filter_default_false(): ph = PromptHandler() ph.load_prompt_dict({"p": "[debug] debug info\nbase"}) - # When no flags are passed at all, _apply_flag_filter is not called, - # so flagged lines are kept as-is (including the tag text after regex sub). - result = ph.prompt_format("p", debug=False) + result = ph.prompt_format("p") assert "debug info" not in result assert "base" in result +def test_flag_filter_defaults_false_with_format_variables(): + ph = PromptHandler() + ph.load_prompt_dict({"p": "[debug] debug {name}\nhello {name}"}) + result = ph.prompt_format("p", name="Alice") + assert result == "hello Alice" + + def test_flag_filter_unflagged_lines_always_kept(): ph = PromptHandler() ph.load_prompt_dict({"p": "line1\nline2\nline3"}) diff --git a/tests/unit/test_source_format_merge.py b/tests/unit/test_source_format_merge.py index 98b7eaca..3d6add31 100644 --- a/tests/unit/test_source_format_merge.py +++ b/tests/unit/test_source_format_merge.py @@ -11,17 +11,17 @@ assert the union content and line order directly. from reme.schema import FileChunk from reme.steps.index._source_format import join_chunk_entries, merge_session_chunk_intervals, render_chunk_entries -_DIALOG_DIR = "session" +_SESSION_DIR = "session" -def _render(chunks: list[FileChunk], dialog_dir: str, **kwargs) -> str: +def _render(chunks: list[FileChunk], session_dir: str, **kwargs) -> str: """Merge session chunks, render entries, then join — mirroring the search steps.""" return join_chunk_entries( - render_chunk_entries(merge_session_chunk_intervals(chunks, dialog_dir), dialog_dir, **kwargs), + render_chunk_entries(merge_session_chunk_intervals(chunks, session_dir), session_dir, **kwargs), ) -def _chunk(start: int, end: int, text: str, score: float = 1.0, path: str = "session/s1.jsonl") -> FileChunk: +def _chunk(start: int, end: int, text: str, score: float = 1.0, path: str = "session/dialog/s1.jsonl") -> FileChunk: return FileChunk(path=path, start_line=start, end_line=end, text=text, scores={"score": score}) @@ -30,7 +30,7 @@ def test_overlapping_session_chunks_merge_into_union_without_duplicates(): a = _chunk(1, 3, "m1\nm2\nm3\n") b = _chunk(3, 5, "m3\nm4\nm5\n") - answer = _render([a, b], _DIALOG_DIR, include_source=False) + answer = _render([a, b], _SESSION_DIR, include_source=False) assert answer == "m1\nm2\nm3\nm4\nm5" @@ -40,7 +40,7 @@ def test_contained_session_chunk_is_absorbed_by_the_larger_range(): big = _chunk(1, 5, "m1\nm2\nm3\nm4\nm5\n") small = _chunk(2, 4, "m2\nm3\nm4\n") - answer = _render([big, small], _DIALOG_DIR, include_source=False) + answer = _render([big, small], _SESSION_DIR, include_source=False) assert answer == "m1\nm2\nm3\nm4\nm5" @@ -50,7 +50,7 @@ def test_adjacent_session_chunks_merge_end_plus_one_equals_next_start(): a = _chunk(1, 3, "m1\nm2\nm3\n") b = _chunk(4, 6, "m4\nm5\nm6\n") - answer = _render([a, b], _DIALOG_DIR, include_source=False) + answer = _render([a, b], _SESSION_DIR, include_source=False) assert answer == "m1\nm2\nm3\nm4\nm5\nm6" @@ -60,17 +60,17 @@ def test_session_chunks_with_a_gap_are_not_merged(): a = _chunk(1, 3, "m1\nm2\nm3\n") b = _chunk(5, 6, "m5\nm6\n") # line 4 missing -> not adjacent - answer = _render([a, b], _DIALOG_DIR, include_source=False) + answer = _render([a, b], _SESSION_DIR, include_source=False) assert answer == "m1\nm2\nm3\n\nm5\nm6" def test_session_chunks_from_different_files_are_not_merged(): """Overlapping ranges in different session files must stay separate.""" - a = _chunk(1, 3, "A1\nA2\nA3\n", path="session/s1.jsonl") - b = _chunk(2, 4, "B2\nB3\nB4\n", path="session/s2.jsonl") + a = _chunk(1, 3, "A1\nA2\nA3\n", path="session/dialog/s1.jsonl") + b = _chunk(2, 4, "B2\nB3\nB4\n", path="session/dialog/s2.jsonl") - answer = _render([a, b], _DIALOG_DIR, include_source=False) + answer = _render([a, b], _SESSION_DIR, include_source=False) assert answer == "A1\nA2\nA3\n\nB2\nB3\nB4" @@ -80,7 +80,7 @@ def test_non_session_chunks_are_never_merged(): a = _chunk(1, 3, "M1\nM2\nM3\n", path="daily/a.md") b = _chunk(2, 4, "M2\nM3\nM4\n", path="daily/a.md") - answer = _render([a, b], _DIALOG_DIR, include_source=False) + answer = _render([a, b], _SESSION_DIR, include_source=False) assert answer == "M1\nM2\nM3\n\nM2\nM3\nM4" @@ -91,7 +91,7 @@ def test_merge_preserves_line_order_regardless_of_input_rank_order(): earlier = _chunk(1, 3, "m1\nm2\nm3\n", score=1.0) # Higher-scored later-range chunk is listed first (as a ranker would). - answer = _render([later, earlier], _DIALOG_DIR, include_source=False) + answer = _render([later, earlier], _SESSION_DIR, include_source=False) assert answer == "m1\nm2\nm3\nm4\nm5" @@ -101,10 +101,10 @@ def test_merged_header_spans_the_union_range_and_keeps_best_score(): a = _chunk(1, 3, "m1\nm2\nm3\n", score=2.0) b = _chunk(3, 5, "m3\nm4\nm5\n", score=7.0) - answer = _render([a, b], _DIALOG_DIR, include_source=True) + answer = _render([a, b], _SESSION_DIR, include_source=True) assert answer.count("==========") == 2 # exactly one header (open + close markers) - assert "session/s1.jsonl:1-5" in answer + assert "session/dialog/s1.jsonl:1-5" in answer assert "score=7.0000" in answer @@ -114,7 +114,7 @@ def test_separate_intervals_in_same_file_stay_separate(): b = _chunk(3, 4, "m3\nm4\n") # adjacent to a -> merges with a into 1-4 c = _chunk(10, 11, "m10\nm11\n") # far away -> separate - answer = _render([a, b, c], _DIALOG_DIR, include_source=False) + answer = _render([a, b, c], _SESSION_DIR, include_source=False) assert answer == "m1\nm2\nm3\nm4\n\nm10\nm11" @@ -123,12 +123,12 @@ def test_same_file_units_stay_adjacent_and_sorted_even_when_interleaved_by_rank( """Two disjoint units of one session file are grouped together and ordered by ``start_line``, even when a different file is ranked between them and the lower interval was ranked last.""" - s1_high = _chunk(10, 12, "S1x\nS1y\nS1z\n", score=9.0, path="session/s1.jsonl") - s2_mid = _chunk(1, 3, "S2a\nS2b\nS2c\n", score=5.0, path="session/s2.jsonl") - s1_low = _chunk(1, 3, "S1a\nS1b\nS1c\n", score=1.0, path="session/s1.jsonl") + s1_high = _chunk(10, 12, "S1x\nS1y\nS1z\n", score=9.0, path="session/dialog/s1.jsonl") + s2_mid = _chunk(1, 3, "S2a\nS2b\nS2c\n", score=5.0, path="session/dialog/s2.jsonl") + s1_low = _chunk(1, 3, "S1a\nS1b\nS1c\n", score=1.0, path="session/dialog/s1.jsonl") # Rank order (as a ranker would emit, by score desc): s1[10-12], s2[1-3], s1[1-3]. - answer = _render([s1_high, s2_mid, s1_low], _DIALOG_DIR, include_source=False) + answer = _render([s1_high, s2_mid, s1_low], _SESSION_DIR, include_source=False) # s1's two units are adjacent and sorted by start_line (1-3 before 10-12), # placed at s1's earliest rank (0), so the whole s1 block precedes s2. @@ -137,13 +137,13 @@ def test_same_file_units_stay_adjacent_and_sorted_even_when_interleaved_by_rank( def test_same_file_units_adjacency_with_source_headers(): """Header view: same-file units are contiguous and ascending; other files follow.""" - s1_high = _chunk(10, 12, "S1x\nS1y\nS1z\n", score=9.0, path="session/s1.jsonl") - s2_mid = _chunk(1, 3, "S2a\nS2b\nS2c\n", score=5.0, path="session/s2.jsonl") - s1_low = _chunk(1, 3, "S1a\nS1b\nS1c\n", score=1.0, path="session/s1.jsonl") + s1_high = _chunk(10, 12, "S1x\nS1y\nS1z\n", score=9.0, path="session/dialog/s1.jsonl") + s2_mid = _chunk(1, 3, "S2a\nS2b\nS2c\n", score=5.0, path="session/dialog/s2.jsonl") + s1_low = _chunk(1, 3, "S1a\nS1b\nS1c\n", score=1.0, path="session/dialog/s1.jsonl") - answer = _render([s1_high, s2_mid, s1_low], _DIALOG_DIR, include_source=True) + answer = _render([s1_high, s2_mid, s1_low], _SESSION_DIR, include_source=True) headers = [line for line in answer.splitlines() if line.startswith("==========")] - assert headers[0].split(" [")[0] == "========== session/s1.jsonl:1-3" - assert headers[1].split(" [")[0] == "========== session/s1.jsonl:10-12" - assert headers[2].split(" [")[0] == "========== session/s2.jsonl:1-3" + assert headers[0].split(" [")[0] == "========== session/dialog/s1.jsonl:1-3" + assert headers[1].split(" [")[0] == "========== session/dialog/s1.jsonl:10-12" + assert headers[2].split(" [")[0] == "========== session/dialog/s2.jsonl:1-3"