scriptoriumadmin/ReMe
1
0
Fork 0
mirror of https://github.com/agentscope-ai/ReMe.git synced 2026-11-06 07:51:11 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 11

Merge remote-tracking branch 'upstream/main' into sandbox

Browse source
This commit is contained in:
酱牛肉 2026-08-05 19:25:02 +08:00
commit dccb602ea7
69 changed files with 3583 additions and 509 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

15
.gitignore vendored
View file

@ -61,17 +61,14 @@ datasets/
# Claude Code skills (local only)
.claude/skills/
# Memory workspaces (keep dirs, ignore contents)
benchmark/memory_workspaces/
# Benchmark memory workspaces (created on demand by run.py via mkdir)
benchmark/*/workspaces/
# LongMemEval raw data (download via download.py)
benchmark/datasets/longmemeval/longmemeval_s_reme_cleaned.json
# Benchmark datasets (LongMemEval via download.py, BEAM via git clone)
benchmark/*/dataset/
# BEAM dataset (clone via git)
benchmark/datasets/BEAM/
# Benchmark outputs
benchmark/results/
# Benchmark outputs (created on demand by run.py via mkdir)
benchmark/*/results/
# integration tests outputs
tests/integration/logs/

2
README.md
View file

@ -59,6 +59,8 @@ memory, then continuously indexes, links, and consolidates that memory for futur
## 📰 News
- [2026.08] - [Experience-driven enhancement method](benchmark/toolmemory/README.md) of agent tool-use
execution built on ReMe is available on [arXiv:2608.03403](https://arxiv.org/abs/2608.03403).
- [2026.07] - Introduced optional Cookbooks: [Daily Paper](cookbook/daily_paper/README.md) for paper discovery and
analysis, and [Auto Fin](cookbook/auto-fin/README.md) for file-native ETF event research based on CLS news and
historical market reactions.

3
README_ZH.md
View file

@ -52,6 +52,9 @@ Agent 能够可靠召回。
## 📰 新闻
- [2026.08] - 基于 ReMe 的智能体工具使用
[经验驱动增强方法](benchmark/toolmemory/README_ZH.md)已发布，见
[arXiv:2608.03403](https://arxiv.org/abs/2608.03403)。
- [2026.07] - 新增可选 Cookbook 工作流：[每日论文](cookbook/daily_paper/README_ZH.md)用于论文发现与解析，
[Auto Fin](cookbook/auto-fin/README_ZH.md)用于结合财联社新闻和历史行情开展文件化 ETF 事件研究。
- [2026.07] -

36
benchmark/README.md
View file

@ -34,22 +34,32 @@ Model names and component wiring live in the ReMe configs referenced by each ben
## 2. Download Datasets
See [`datasets/README_EN.md`](datasets/README_EN.md) for full details.
Each benchmark keeps its own data under its directory:
`benchmark/<name>/dataset` (input data), `benchmark/<name>/workspaces`
(per-item memory workspaces), and `benchmark/<name>/results` (evaluation outputs).
All three are excluded from Git.
**LongMemEval** (downloaded from a HuggingFace mirror):
**LongMemEval** — ReMe uses only the **cleaned-S** split, hosted on HuggingFace:
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)
(the script downloads via the hf-mirror.com mirror; to use a different mirror,
modify `BASE_URL` in `download.py`):
```bash
cd benchmark/datasets/longmemeval
python download.py # downloads the cleaned-S dataset; skips if already present
cd benchmark/longmemeval
python download.py # saves dataset/longmemeval_s_reme_cleaned.json; skips if already present
```
**BEAM** (public repository, cloned into `benchmark/datasets/`):
**BEAM** (public repository, cloned into `benchmark/beam/dataset/`):
```bash
cd benchmark/datasets
mkdir -p benchmark/beam/dataset
cd benchmark/beam/dataset
git clone https://github.com/mohammadtavakoli78/BEAM.git
```
After cloning, `benchmark/beam/dataset/BEAM/` should contain `chats/`, `src/`,
`topics/` and other subdirectories.
## 3. Run LongMemEval
From the repository root:
@ -78,12 +88,12 @@ python benchmark/longmemeval/run.py --eval_only # reuse existing w
| `dataset.path` | Dataset file to evaluate (e.g. `longmemeval_s_reme_cleaned.json`); ground truth is included. |
| `dataset.start_index` / `num_items` | Slice of items to evaluate. |
| `dataset.question_types` | Filter by question type; empty = all. |
| `dataset.workspace_root` | Per-item workspace root (`benchmark/memory_workspaces/longmemeval-s`). |
| `dataset.workspace_root` | Per-item workspace root (`benchmark/longmemeval/workspaces/longmemeval-s`). |
| `evaluation.num_workers` | `0` = auto (cpu-2), `1` = sequential, `>1` = parallel. |
| `evaluation.filter_future_sessions` | Only ingest sessions with timestamp ≤ `question_date`. |
| `reme.config` | ReMe config used (`lme.yaml`). |
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | Dream triggering behavior. |
| `output.dir` | Results directory (`benchmark/results/longmemeval`). |
| `output.dir` | Results directory (`benchmark/longmemeval/results`). |
## 4. Run BEAM
@ -107,14 +117,14 @@ python benchmark/beam/run.py --eval_only # reuse existing workspac
| Key | Meaning |
| --- | --- |
| `dataset.beam_root` | BEAM dataset root (`benchmark/datasets/BEAM`). |
| `dataset.beam_root` | BEAM dataset root (`benchmark/beam/dataset/BEAM`). |
| `dataset.chat_size` | Variant to run: `100K` / `500K` / `1M` / `10M`. |
| `dataset.case_ids` | Specific cases (e.g. `["1","2"]`); empty = all cases. |
| `dataset.start_index` / `num_items` | Case pagination (`num_items` `0` = all). |
| `dataset.workspace_root` | Per-case workspace root (`benchmark/memory_workspaces/beam`). |
| `dataset.workspace_root` | Per-case workspace root (`benchmark/beam/workspaces/beam`). |
| `evaluation.num_workers` | `0` = auto, `1` = sequential, `>1` = parallel. |
| `reme.config` | ReMe config used (`beam.yaml`). |
| `output.dir` | Results directory (`benchmark/results/beam`). |
| `output.dir` | Results directory (`benchmark/beam/results`). |
## 5. Outputs & Logs
@ -141,5 +151,5 @@ The script gracefully sends `SIGTERM` to the whole process tree, then escalates
Recorded evaluation results are available in:
- [`result-longmemeval.md`](./result-longmemeval.md)
- [`result-beam.md`](./result-beam.md)
- [`longmemeval.md`](./results_md/longmemeval.md)
- [`beam.md`](./results_md/beam.md)

33
benchmark/README_ZH.md
View file

@ -32,22 +32,29 @@ EMBEDDING_BASE_URL=...
## 2. 下载数据集
完整说明见 [`datasets/README.md`](datasets/README.md)。
每个基准的数据都存放在各自目录下：`benchmark/<name>/dataset`（输入数据）、
`benchmark/<name>/workspaces`（按条目隔离的记忆工作区）、
`benchmark/<name>/results`（评测输出）。三者均不纳入 Git 版本管理。
**LongMemEval**（从 HuggingFace 镜像下载）：
**LongMemEval** —— ReMe 仅使用 **cleaned-S** 版本，数据托管在 HuggingFace：
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)
（下载脚本经 hf-mirror.com 镜像源获取，如需更换源请修改 `download.py` 中的 `BASE_URL`）：
```bash
cd benchmark/datasets/longmemeval
python download.py # 下载 cleaned-S 数据文件，已存在则自动跳过
cd benchmark/longmemeval
python download.py # 保存为 dataset/longmemeval_s_reme_cleaned.json，已存在则自动跳过
```
**BEAM**（公开仓库，clone 到 `benchmark/datasets/` 下）：
**BEAM**（公开仓库，clone 到 `benchmark/beam/dataset/` 下）：
```bash
cd benchmark/datasets
mkdir -p benchmark/beam/dataset
cd benchmark/beam/dataset
git clone https://github.com/mohammadtavakoli78/BEAM.git
```
clone 完成后，`benchmark/beam/dataset/BEAM/` 目录下应包含 `chats/`、`src/`、`topics/` 等子目录。
## 3. 运行 LongMemEval
在仓库根目录执行：
@ -76,12 +83,12 @@ python benchmark/longmemeval/run.py --eval_only # 复用已有工
| `dataset.path` | 待评测的数据集文件（如 `longmemeval_s_reme_cleaned.json`），已包含 ground truth。 |
| `dataset.start_index` / `num_items` | 评测条目的切片范围。 |
| `dataset.question_types` | 按问题类型过滤，空表示全部。 |
| `dataset.workspace_root` | 条目工作区根目录（`benchmark/memory_workspaces/longmemeval-s`）。 |
| `dataset.workspace_root` | 条目工作区根目录（`benchmark/longmemeval/workspaces/longmemeval-s`）。 |
| `evaluation.num_workers` | `0` = 自动（cpu-2），`1` = 串行，`>1` = 并行。 |
| `evaluation.filter_future_sessions` | 仅摄入时间戳 ≤ `question_date` 的会话。 |
| `reme.config` | 使用的 ReMe 配置（`lme.yaml`）。 |
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | dream 触发行为。 |
| `output.dir` | 结果目录（`benchmark/results/longmemeval`）。 |
| `output.dir` | 结果目录（`benchmark/longmemeval/results`）。 |
## 4. 运行 BEAM
@ -105,14 +112,14 @@ python benchmark/beam/run.py --eval_only # 复用已有工作区
| 配置项 | 含义 |
| --- | --- |
| `dataset.beam_root` | BEAM 数据集根目录（`benchmark/datasets/BEAM`）。 |
| `dataset.beam_root` | BEAM 数据集根目录（`benchmark/beam/dataset/BEAM`）。 |
| `dataset.chat_size` | 运行的变体：`100K` / `500K` / `1M` / `10M`。 |
| `dataset.case_ids` | 指定 case（如 `["1","2"]`），空表示全部。 |
| `dataset.start_index` / `num_items` | case 分页（`num_items` 为 `0` 表示全部）。 |
| `dataset.workspace_root` | case 工作区根目录（`benchmark/memory_workspaces/beam`）。 |
| `dataset.workspace_root` | case 工作区根目录（`benchmark/beam/workspaces/beam`）。 |
| `evaluation.num_workers` | `0` = 自动，`1` = 串行，`>1` = 并行。 |
| `reme.config` | 使用的 ReMe 配置（`beam.yaml`）。 |
| `output.dir` | 结果目录（`benchmark/results/beam`）。 |
| `output.dir` | 结果目录（`benchmark/beam/results`）。 |
## 5. 输出与日志
@ -138,5 +145,5 @@ bash benchmark/kill.sh <PID>
已记录的评测结果见：
- [`result-longmemeval.md`](./result-longmemeval.md)
- [`result-beam.md`](./result-beam.md)
- [`longmemeval.md`](./results_md/longmemeval.md)
- [`beam.md`](./results_md/beam.md)

11
benchmark/beam/config.yaml
View file

@ -2,21 +2,22 @@
# This file controls what/how to evaluate.
dataset:
beam_root: "benchmark/datasets/BEAM" # BEAM dataset root
chat_size: "1M" # 100K | 500K | 1M | 10M (dataset variant)
beam_root: "benchmark/beam/dataset/BEAM" # BEAM dataset root
chat_size: "1M" # 100K | 500K | 1M | 10M (dataset variant)
case_ids: [] # empty = all cases; or ["1", "2", "3"]
start_index: 0 # first case index (for pagination)
num_items: 0 # 0 = all cases; >0 = limit
workspace_root: "benchmark/memory_workspaces/beam" # workspace root for case workspaces
workspace_root: "benchmark/beam/workspaces/beam" # workspace root for case workspaces
evaluation:
num_workers: 18 # 0 = auto; 1 = sequential; >1 = parallel (per-case)
num_workers: 20 # 0 = auto; 1 = sequential; >1 = parallel (per-case)
compress_session: false # true = compress session chunks in search_v2 (query-aware); false = no compression
reme:
config: "beam.yaml" # reme config (in reme/config/)
output:
dir: "benchmark/results/beam"
dir: "benchmark/beam/results"
log_dir: "logs" # log directory (relative to project root)
log_prefix: "beam" # benchmark name used in log filenames
log_to_console: true

65
benchmark/beam/run.py
View file

@ -33,7 +33,7 @@ _PROJECT_ROOT = Path(__file__).parent.parent.parent
load_dotenv(_PROJECT_ROOT / ".env")
# Workspace root — read from config.yaml (dataset.workspace_root)
_WORKSPACE_ROOT_DEFAULT = "benchmark/memory_workspaces/beam"
_WORKSPACE_ROOT_DEFAULT = "benchmark/beam/workspaces/beam"
# ---------------------------------------------------------------------------
# Logging
@ -247,7 +247,7 @@ def get_available_cases(beam_root: Path, chat_size: str) -> list[str]:
# ---------------------------------------------------------------------------
# Answer generation
# ---------------------------------------------------------------------------
async def answer_question_agentic(app, question: str) -> tuple[str, dict]:
async def answer_question_agentic(app, question: str, compress_session: bool = False) -> tuple[str, dict]:
"""Answer a probing question using ReMe's agentic_answer job.
Returns (answer, metadata)
@ -264,6 +264,7 @@ async def answer_question_agentic(app, question: str) -> tuple[str, dict]:
query_resp = await app.run_job(
"agentic_answer",
query=question,
compress_session=compress_session,
)
answer = (query_resp.answer or "").strip()
@ -323,7 +324,8 @@ async def evaluate_case(eval_config: dict, case_id: str, eval_only: bool = False
dataset_cfg = eval_config["dataset"]
chat_size = dataset_cfg["chat_size"]
beam_root = _PROJECT_ROOT / dataset_cfg.get("beam_root", "benchmark/datasets/BEAM")
compress_session = bool(eval_config["evaluation"].get("compress_session", False))
beam_root = _PROJECT_ROOT / dataset_cfg.get("beam_root", "benchmark/beam/dataset/BEAM")
chat_path = beam_root / "chats" / chat_size / case_id / "chat.json"
probing_questions_path = beam_root / "chats" / chat_size / case_id / "probing_questions" / "probing_questions.json"
@ -384,13 +386,20 @@ async def evaluate_case(eval_config: dict, case_id: str, eval_only: bool = False
app = Application(**cfg)
await app.start()
from reme.utils.evaluation_interface import check_agent_token_usage # noqa: E402
_MEM_AGENT_NAMES = ("default", "bench")
sessions_ingested = 0
memory_token_usage: dict[str, dict[str, int | None]] = {}
try:
if not eval_only:
# ── Phase 1: Ingest sessions ──────────────────────────────
# ── Phase 1: Ingest sessions (with token tracking) ─────────
sessions = load_beam_chat(chat_path, chat_size, case_id)
logger.info(f"[Case {case_id}] Loaded {len(sessions)} sessions from chat.json")
# Snapshot token counters before memory construction
mem_token_start = {name: check_agent_token_usage(name, app.context) for name in _MEM_AGENT_NAMES}
for i, session in enumerate(sessions):
logger.info(
f"[Case {case_id}] Ingesting session {i+1}/{len(sessions)}: "
@ -417,6 +426,17 @@ async def evaluate_case(eval_config: dict, case_id: str, eval_only: bool = False
await app.run_job("digest_update")
logger.info(f"[Case {case_id}] Ingestion complete.")
# Compute memory construction token deltas
for name in _MEM_AGENT_NAMES:
end_usage = check_agent_token_usage(name, app.context)
delta: dict[str, int | None] = {}
for metric in _TOKEN_USAGE_METRICS:
current = end_usage[metric]
start = mem_token_start[name][metric]
delta[metric] = None if current is None else current - (start or 0)
memory_token_usage[name] = delta
logger.info(f"[Case {case_id}] Memory construction token usage: {memory_token_usage}")
# ── Phase 2: Answer + Judge probing questions ───────────────
with open(probing_questions_path, encoding="utf-8") as f:
probing_questions = json.load(f)
@ -452,6 +472,7 @@ async def evaluate_case(eval_config: dict, case_id: str, eval_only: bool = False
agentic_answer, agentic_meta = await answer_question_agentic(
app,
question,
compress_session=compress_session,
)
except Exception as e:
logger.error(f"[Case {case_id}] Agentic answer failed: {e}")
@ -494,6 +515,7 @@ async def evaluate_case(eval_config: dict, case_id: str, eval_only: bool = False
"sessions_ingested": sessions_ingested,
"total_questions": len(all_question_results),
"questions": all_question_results,
"memory_token_usage": memory_token_usage,
}
@ -561,7 +583,7 @@ def main( # pylint: disable=too-many-statements
setup_logging(log_level, reme_log_level, log_dir=log_dir_abs)
dataset_cfg = eval_config["dataset"]
chat_size = dataset_cfg["chat_size"]
beam_root = _PROJECT_ROOT / dataset_cfg.get("beam_root", "benchmark/datasets/BEAM")
beam_root = _PROJECT_ROOT / dataset_cfg.get("beam_root", "benchmark/beam/dataset/BEAM")
# Determine which cases to run
case_ids = dataset_cfg.get("case_ids") or []
@ -593,7 +615,7 @@ def main( # pylint: disable=too-many-statements
logger.info(f"Using {num_workers} worker(s)")
# Create output directory
output_dir = _PROJECT_ROOT / output_cfg.get("dir", "benchmark/results/beam")
output_dir = _PROJECT_ROOT / output_cfg.get("dir", "benchmark/beam/results")
output_dir.mkdir(parents=True, exist_ok=True)
# Create workspace root directory
@ -697,10 +719,14 @@ def main( # pylint: disable=too-many-statements
all_binary_scores: list[float] = []
all_tool_call_totals: list[int] = []
all_token_usages: list[dict[str, int | None]] = []
all_memory_token_usages: list[dict[str, dict[str, int | None]]] = []
for case_result in results:
if "error" in case_result:
continue
mem_usage = case_result.get("memory_token_usage", {})
if mem_usage:
all_memory_token_usages.append(mem_usage)
for q in case_result.get("questions", []):
judgment = q.get("agentic_judgment", {})
score = judgment.get("llm_judge_score", 0.0)
@ -723,6 +749,26 @@ def main( # pylint: disable=too-many-statements
all_tool_call_totals.append(sum(metadata.get("tool_counts", {}).values()))
all_token_usages.append(metadata.get("token_usage", {}))
# Memory construction token usage summary
if all_memory_token_usages:
print("\n ── Memory Construction Token Usage ──")
for agent_name in ("default", "bench"):
for metric in _TOKEN_USAGE_METRICS:
values = [
usage[agent_name][metric]
for usage in all_memory_token_usages
if usage.get(agent_name, {}).get(metric) is not None
]
if values:
total = sum(values)
mean, std = _mean_and_std(values)
print(
f" {agent_name}/{metric}: total={total} mean={mean:.2f} std={std:.2f} ({len(values)} cases)",
)
else:
print(f" {agent_name}/{metric}: unavailable")
print()
print("\n ── AGENTIC ──")
if all_scores:
for qtype in sorted(type_scores.keys()):
@ -757,7 +803,14 @@ def main( # pylint: disable=too-many-statements
continue
n_qs = case_result.get("total_questions", 0)
n_sessions = case_result.get("sessions_ingested", 0)
mem_usage = case_result.get("memory_token_usage", {})
parts = [f"Case {case_id}: {n_sessions} sessions, {n_qs} questions"]
# Append memory construction total tokens if available
for agent_name in ("default", "bench"):
agent_usage = mem_usage.get(agent_name, {})
total = agent_usage.get("total_tokens")
if total is not None:
parts.append(f"mem_{agent_name}_tokens={total}")
questions = case_result.get("questions", [])
scores = [q.get("agentic_judgment", {}).get("llm_judge_score", 0.0) for q in questions]
if scores:

40
benchmark/datasets/README.md
View file

@ -1,40 +0,0 @@
# 数据集下载说明
本目录包含 ReMe 评测所需的数据集。部分数据集体积较大，不纳入 Git 版本管理，需要手动下载。
## LongMemEval（cleaned-S）
ReMe 仅使用 LongMemEval 的 **cleaned-S** 版本，数据托管在 HuggingFace：
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)
（下载脚本经 hf-mirror.com 镜像源获取）。
按以下步骤下载：
```bash
cd benchmark/datasets/longmemeval
# 下载 cleaned-S 数据文件（已存在则自动跳过）
python download.py
```
下载完成后，目录下应包含以下文件：
| 文件名 | 说明 |
| --- | --- |
| `longmemeval_s_reme_cleaned.json` | cleaned-S 数据集，已包含 ground truth 字段 |
| `download.py` | 下载脚本（已随仓库提供） |
> **注意**：下载脚本使用 hf-mirror.com 镜像源，如需更换源请修改 `download.py` 中的 `BASE_URL`。
下载完成后即可参照 [`benchmark/README_ZH.md`](../README_ZH.md) 运行 LongMemEval 评测。
## BEAM
BEAM 数据集为公开仓库，直接 clone 到 `benchmark/datasets/` 目录下即可：
```bash
cd benchmark/datasets
git clone https://github.com/mohammadtavakoli78/BEAM.git
```
clone 完成后，`benchmark/datasets/BEAM/` 目录下应包含 `chats/`、`src/`、`topics/` 等子目录。

40
benchmark/datasets/README_EN.md
View file

@ -1,40 +0,0 @@
# Dataset Download Guide
This directory contains datasets required for ReMe evaluation. Some datasets are large and excluded from Git version control — they must be downloaded manually.
## LongMemEval (cleaned-S)
ReMe uses only the **cleaned-S** split of LongMemEval, hosted on HuggingFace:
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)
(the script downloads via the hf-mirror.com mirror).
Download it with:
```bash
cd benchmark/datasets/longmemeval
# Download the cleaned-S data file (skipped automatically if it already exists)
python download.py
```
After downloading, the directory should contain:
| File | Description |
| --- | --- |
| `longmemeval_s_reme_cleaned.json` | cleaned-S dataset with ground truth fields included |
| `download.py` | Download script (included in repo) |
> **Note**: The download script uses hf-mirror.com by default. To use a different mirror, modify `BASE_URL` in `download.py`.
Once the download completes, follow [`benchmark/README.md`](../README.md) to run the LongMemEval evaluation.
## BEAM
BEAM is a public repository. Clone it directly into the `benchmark/datasets/` directory:
```bash
cd benchmark/datasets
git clone https://github.com/mohammadtavakoli78/BEAM.git
```
After cloning, `benchmark/datasets/BEAM/` should contain `chats/`, `src/`, `topics/` and other subdirectories.

7
benchmark/longmemeval/config.yaml
View file

@ -2,12 +2,12 @@
# This file controls what/how to evaluate.
dataset:
path: "benchmark/datasets/longmemeval/longmemeval_s_reme_cleaned.json"
path: "benchmark/longmemeval/dataset/longmemeval_s_reme_cleaned.json"
start_index: 0 # first item index
num_items: 500 # how many items to evaluate (starting from start_index)
max_sessions: 0 # 0 = all sessions; >0 = limit sessions per item for testing
question_types: [] # filter by question_type; empty list = no filtering (all types)
workspace_root: "benchmark/memory_workspaces/longmemeval-s" # workspace root for item workspaces
workspace_root: "benchmark/longmemeval/workspaces/longmemeval-s" # workspace root for item workspaces
evaluation:
# LLM-as-judge uses the 'judge' as_llm component defined in lme.yaml
@ -15,6 +15,7 @@ evaluation:
# Judgment is always binary (yes/no) — defined in lme/llm_judge.yaml
num_workers: 32 # 0 = auto (cpu_count - 2, min 1); 1 = sequential; >1 = parallel
filter_future_sessions: true # true = only ingest sessions with timestamp <= question_date
compress_session: false # true = compress session chunks in search_v2 (query-aware); false = no compression
reme:
config: "lme.yaml" # reme config to use (in reme/config/)
@ -25,7 +26,7 @@ reme:
dream_max_units: 5
output:
dir: "benchmark/results/longmemeval"
dir: "benchmark/longmemeval/results"
log_dir: "logs" # log directory (relative to project root)
log_prefix: "longmemeval" # benchmark name used in log filenames
log_to_console: true

8
benchmark/datasets/longmemeval/download.py → benchmark/longmemeval/download.py
View file

@ -3,8 +3,9 @@
Source: https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2
(downloaded via the hf-mirror.com mirror for reliability).
The file ``longmemeval_s_reme_cleaned.json`` is saved under this directory using the same
name as on the remote (``benchmark/longmemeval/config.yaml`` points to it).
The file ``longmemeval_s_reme_cleaned.json`` is saved under ``dataset/`` next to this
script using the same name as on the remote (``benchmark/longmemeval/config.yaml``
points to it).
Usage:
python download.py # download cleaned-S (skip if it already exists)
@ -15,7 +16,7 @@ import sys
import urllib.request
BASE_URL = "https://hf-mirror.com/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2/resolve/main"
TARGET_DIR = os.path.dirname(os.path.abspath(__file__))
TARGET_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "dataset")
# Files to download (saved with the same name as on the remote).
FILES = [
@ -59,6 +60,7 @@ def _progress(block_num, block_size, total_size):
if __name__ == "__main__":
os.makedirs(TARGET_DIR, exist_ok=True)
print(f"Downloading LongMemEval cleaned-S dataset to: {TARGET_DIR}\n")
for fname in FILES:
download_file(fname)

12
benchmark/longmemeval/run.py
View file

@ -32,7 +32,7 @@ _PROJECT_ROOT = Path(__file__).parent.parent.parent
load_dotenv(_PROJECT_ROOT / ".env")
# Workspace root for evaluation items — read from config.yaml (dataset.workspace_root)
_WORKSPACE_ROOT_DEFAULT = "benchmark/memory_workspaces/longmemeval-s"
_WORKSPACE_ROOT_DEFAULT = "benchmark/longmemeval/workspaces/longmemeval-s"
# ---------------------------------------------------------------------------
# Logging
@ -426,6 +426,7 @@ async def evaluate_item(item: dict, eval_config: dict, item_index: int, eval_onl
# ── Phase 4: Ask question via agentic_answer job (ReAct agent) ──
question = item["question"]
compress_session = bool(eval_config["evaluation"].get("compress_session", False))
question_date_raw = item.get("question_date", "")
question_dt = parse_haystack_date(question_date_raw) if question_date_raw else None
query_time = to_iso(question_dt) if question_dt else ""
@ -440,7 +441,12 @@ async def evaluate_item(item: dict, eval_config: dict, item_index: int, eval_onl
app.context,
) as token_usages,
):
query_resp = await app.run_job("agentic_answer", query=question, query_time=query_time)
query_resp = await app.run_job(
"agentic_answer",
query=question,
query_time=query_time,
compress_session=compress_session,
)
agentic_tool_counts = tool_counts
agentic_token_usage = token_usages["bench"]
agentic_response = (query_resp.answer or "").strip()
@ -593,7 +599,7 @@ def main(
logger.info(f"Using {num_workers} worker(s)")
# Create output directory
output_dir = _PROJECT_ROOT / output_cfg.get("dir", "benchmark/results/longmemeval")
output_dir = _PROJECT_ROOT / output_cfg.get("dir", "benchmark/longmemeval/results")
output_dir.mkdir(parents=True, exist_ok=True)
# Create workspace root directory

36
benchmark/result-beam.md
View file

@ -1,36 +0,0 @@
# beam result
## longmemeval版本的prompt
### 100K
| 题型 | Prompted(limit=15) | Agentic |
|---|---|---|
| abstention | 0.525 | 0.575 |
| contradiction_resolution | 0.100 | 0.384 |
| event_ordering | 0.403 | 0.465 |
| information_extraction | 0.618 | 0.884 |
| instruction_following | 0.481 | 0.719 |
| knowledge_update | 0.637 | 0.650 |
| multi_session_reasoning | 0.444 | 0.633 |
| preference_following | 0.706 | 0.829 |
| summarization | 0.423 | 0.617 |
| temporal_reasoning | 0.344 | 0.550 |
| **OVERALL** | **0.468** | **0.631** |
### 1M
| 题型 | Prompted(limit=15) | Prompted Binary | Agentic | Agentic Binary |
|---|---|---|---|---|
| abstention | 0.464 | 0.464 | 0.514 | 0.514 |
| contradiction_resolution | 0.079 | 0.068 | 0.373 | 0.339 |
| event_ordering | 0.455 | 0.334 | 0.547 | 0.450 |
| information_extraction | 0.653 | 0.589 | 0.818 | 0.764 |
| instruction_following | 0.541 | 0.524 | 0.765 | 0.745 |
| knowledge_update | 0.571 | 0.507 | 0.636 | 0.629 |
| multi_session_reasoning | 0.426 | 0.324 | 0.593 | 0.540 |
| preference_following | 0.718 | 0.676 | 0.838 | 0.824 |
| summarization | 0.516 | 0.303 | 0.661 | 0.478 |
| temporal_reasoning | 0.198 | 0.169 | 0.394 | 0.383 |
| **OVERALL** | **0.462** | **0.396** | **0.614** | **0.567** |

54
benchmark/results_md/beam.md Normal file
View file

@ -0,0 +1,54 @@
# beam result
## longmemeval版本的prompt
### 100K
agentscope==2.0.4.post1, conda reme 环境, 20 并发, eval-only（复用已构建 memory）
(2026-08-05, 20 cases / 400 Qs, 总耗时 46.0 min)
| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|---|---|---|---|---|---|---|
| abstention | 0.550 | 0.550 | 96,031 | 1,070 | 97,101 | 4.58 |
| contradiction_resolution | 0.438 | 0.412 | 32,263 | 872 | 33,135 | 2.48 |
| event_ordering | 0.501 | 0.423 | 140,195 | 5,163 | 145,358 | 4.70 |
| information_extraction | 0.873 | 0.832 | 50,245 | 883 | 51,128 | 3.15 |
| instruction_following | 0.750 | 0.725 | 37,986 | 848 | 38,834 | 2.67 |
| knowledge_update | 0.688 | 0.675 | 31,198 | 651 | 31,849 | 2.27 |
| multi_session_reasoning | 0.626 | 0.584 | 85,038 | 4,563 | 89,601 | 4.28 |
| preference_following | 0.925 | 0.912 | 34,281 | 989 | 35,270 | 2.50 |
| summarization | 0.623 | 0.461 | 89,657 | 2,056 | 91,713 | 4.12 |
| temporal_reasoning | 0.637 | 0.625 | 34,563 | 1,049 | 35,612 | 2.52 |
| **OVERALL** | **0.661** | **0.620** | **63,146** | **1,814** | **64,960** | **3.33** |
Memory Construction 平均 token 消耗（default agent, 20 cases 全量构建）:
| Agent | input tok/case | output tok/case | total tok/case |
|---|---|---|---|
| default | 2,172,316 | 136,697 | 2,309,013 |
### 1M
agentscope==2.0.4.post1, conda reme 环境, 20 并发, 全量构建 memory
(2026-08-05, 35 cases / 700 Qs, 总耗时 459.2 min)
| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q |
|---|---|---|---|---|---|---|
| abstention | 0.429 | 0.429 | 118,707 | 1,178 | 119,886 | 4.20 |
| contradiction_resolution | 0.391 | 0.364 | 49,787 | 810 | 50,597 | 2.50 |
| event_ordering | 0.558 | 0.456 | 201,514 | 3,889 | 205,403 | 4.79 |
| information_extraction | 0.809 | 0.772 | 78,950 | 894 | 79,844 | 3.00 |
| instruction_following | 0.852 | 0.832 | 55,757 | 924 | 56,681 | 2.81 |
| knowledge_update | 0.779 | 0.771 | 45,981 | 665 | 46,646 | 2.37 |
| multi_session_reasoning | 0.658 | 0.612 | 138,133 | 2,873 | 141,006 | 4.40 |
| preference_following | 0.798 | 0.777 | 51,796 | 920 | 52,716 | 2.53 |
| summarization | 0.693 | 0.537 | 158,794 | 2,905 | 161,700 | 4.44 |
| temporal_reasoning | 0.536 | 0.536 | 100,176 | 3,148 | 103,324 | 3.90 |
| **OVERALL** | **0.650** | **0.609** | **99,959** | **1,821** | **101,780** | **3.49** |
Memory Construction 平均 token 消耗（default agent, 35 cases 全量构建）:
| Agent | input tok/case | output tok/case | total tok/case |
|---|---|---|---|
| default | 31,943,817 | 1,417,061 | 33,360,878 |

0
benchmark/result-longmemeval.md → benchmark/results_md/longmemeval.md
View file

98
benchmark/toolmemory/README.md Normal file
View file

@ -0,0 +1,98 @@
## Towards Robust Tool Use in Agents via Experience-Driven Adaptive Guidance
**Language**: English (default) / [中文](./README_ZH.md)
> Paper: [arXiv:2608.03403](https://arxiv.org/abs/2608.03403)
> Code: [https://github.com/WangCan1178/ExpG](https://github.com/WangCan1178/ExpG)
<p align="center">
<img src="gitcha.png" alt="ExpG challenges and overview" width="85%">
</p>
### Overview
This folder archives **ExpG**, a tool-use enhancement built on [Agentscope ReMe](https://github.com/agentscope-ai/ReMe). ExpG mines, distills, and reuses experience from historical tool calls to provide **capability boundaries** and **best-practice guidance**, which helps agents:
- Select and invoke tools more robustly under dynamic or noisy environments;
- Let smaller models with guidance outperform larger, memoryless baselines;
- Improve consistently across tool selection, tool calling, and response generation.
**How ReMe is used:** Start the Tool Memory service; historical tool calls are written and evaluated via `add_tool_call_result`, distilled into tool-level guidance via `summary_tool_memory`, then retrieved and injected into later reasoning via `retrieve_tool_memory`. ReMe provides the vector store and service APIs; the acquisition / distillation / reuse strategy is implemented by ExpG. Full implementation and experiments are in [WangCan1178/ExpG](https://github.com/WangCan1178/ExpG).
---
### ExpG Mechanism
ExpG treats tool invocations as learnable experience and runs a three-stage pipeline:
1. **Experience Acquisition**
- Analyze invocation quality from historical trajectories (success/failure, cost, latency, etc.);
- Build structured experience units per tool, recording context, parameter patterns, and outcomes.
2. **Experience Distillation**
- Filter noisy or unhelpful experiences and keep representative patterns;
- Aggregate by equivalence classes to cover common and rare failure modes;
- Summarize with an LLM into generalizable textual guidance.
3. **Experience Reuse**
- Retrieve relevant experience / guidance for future tasks;
- Inject guidance into tool selection, argument generation, and response synthesis;
- Improve stability under dynamic environments and imperfect feedback.
---
### Main Results
Performance comparison (%) across MetaTool, API-Bank, and BFCL-V3. **Bold** indicates the best results within each model.
| Model | Method | MetaTool Pass@1 | MetaTool Avg@3 | MetaTool Pass@3 | API-Bank Pass@1 | API-Bank Avg@3 | API-Bank Pass@3 | BFCL-V3 Pass@1 | BFCL-V3 Avg@3 | BFCL-V3 Pass@3 | Total Pass@1 | Total Avg@3 | Total Pass@3 |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| GPT-5 nano | No Method | 72.62 | 72.76 | 78.49 | 82.96 | 83.46 | 86.97 | 53.80 | 53.00 | 60.95 | 70.82 | 70.62 | 76.63 |
| GPT-5 nano | Few-shot | 74.12 | 75.11 | 82.32 | 83.71 | 83.96 | **87.22** | 56.18 | 55.24 | 61.39 | 72.36 | 72.65 | 79.28 |
| GPT-5 nano | DRAFT | 73.94 | 73.04 | 78.97 | 84.21 | 83.46 | **87.22** | 57.27 | 57.27 | 62.26 | 72.52 | 71.58 | 77.23 |
| GPT-5 nano | Mem0 | 74.96 | 76.13 | 82.92 | 84.96 | 85.21 | **87.22** | 60.95 | 61.61 | 65.08 | 73.98 | 74.67 | 80.35 |
| GPT-5 nano | **ExpG** | **81.67** | **82.07** | **84.60** | **86.72** | **86.55** | **87.22** | **64.43** | **63.99** | **66.38** | **79.32** | **79.22** | **81.69** |
| DeepSeek-V3 | No Method | 83.10 | 82.94 | 84.66 | 84.71 | 84.38 | 85.46 | 58.79 | 59.65 | 65.94 | 78.92 | 78.66 | 81.37 |
| DeepSeek-V3 | Few-shot | 82.74 | 83.90 | 86.28 | 85.21 | 84.63 | 86.22 | 60.52 | 60.30 | 67.90 | 79.08 | 79.45 | 82.92 |
| DeepSeek-V3 | DRAFT | 80.23 | 80.79 | 82.44 | 84.96 | 85.63 | 86.47 | 62.26 | 61.61 | 68.55 | 77.70 | 77.80 | 80.54 |
| DeepSeek-V3 | Mem0 | 83.88 | 84.56 | 86.40 | 85.46 | 85.55 | 86.47 | 65.08 | 65.15 | 68.33 | 80.70 | 80.91 | 83.12 |
| DeepSeek-V3 | **ExpG** | **85.26** | **85.38** | **86.52** | **87.72** | **87.39** | **87.97** | **69.41** | **69.92** | **72.02** | **82.76** | **82.61** | **84.11** |
| Qwen3-8B | No Method | 76.51 | 76.97 | 77.71 | 83.96 | 83.88 | 84.21 | 58.79 | 58.28 | 60.30 | 74.46 | 74.41 | 75.56 |
| Qwen3-8B | Few-shot | 79.93 | 79.83 | 82.92 | 83.71 | 82.62 | 84.96 | 60.09 | 59.29 | 61.39 | 76.91 | 76.27 | 79.32 |
| Qwen3-8B | DRAFT | 78.19 | 77.33 | 77.89 | 85.71 | 84.96 | 85.46 | 60.74 | 60.30 | 62.91 | 76.20 | 75.18 | 76.35 |
| Qwen3-8B | Mem0 | 75.07 | 75.47 | 82.38 | 86.22 | 86.05 | 86.47 | 63.34 | 64.93 | 66.16 | 74.69 | 74.98 | 80.07 |
| Qwen3-8B | **ExpG** | **83.52** | **84.88** | **85.08** | **86.47** | **87.89** | **87.97** | **67.46** | **66.96** | **68.33** | **81.06** | **81.82** | **82.48** |
| Qwen3-32B | No Method | 80.05 | 79.43 | 80.17 | 84.71 | 84.88 | 85.21 | 65.15 | 65.08 | 66.16 | 78.05 | 77.55 | 78.41 |
| Qwen3-32B | **ExpG** | **84.68** | **85.02** | **86.28** | **86.97** | **87.30** | **87.72** | **70.72** | **71.01** | **73.32** | **82.48** | **82.56** | **84.14** |
| Qwen3-235B | No Method | 78.25 | 79.23 | 80.29 | 85.46 | 85.46 | 85.71 | 71.37 | 71.15 | 73.54 | 78.13 | 78.49 | 79.91 |
| Qwen3-235B | **ExpG** | **86.34** | **86.70** | **86.94** | **87.47** | **86.97** | **88.22** | **79.61** | **78.52** | **80.04** | **85.29** | **84.98** | **85.69** |
---
### Reference Code
| Path | Role |
| --- | --- |
| [`tool_memory.py`](./tool_memory.py) | HTTP client for official ReMe Tool Memory APIs (`add_tool_call_result` / `summary_tool_memory` / `retrieve_tool_memory`) |
| [`parse_tool_call_result_prompt.yaml`](./parse_tool_call_result_prompt.yaml) | Prompt for multi-aspect evaluation of each tool call |
| [`summary_tool_memory_prompt.yaml`](./summary_tool_memory_prompt.yaml) | Prompt for summarizing tool call history into guidance |
| [`tool_memory_flows.yaml`](./tool_memory_flows.yaml) | Tool Memory flow / op config excerpt |
These are reference snippets. For the full runnable codebase, see [WangCan1178/ExpG](https://github.com/WangCan1178/ExpG).
---
### Citation
```bibtex
@misc{wang2026expg,
title = {Towards Robust Tool Use in Agents via Experience-Driven Adaptive Guidance},
author = {Can Wang and Haoran Chen and Li Yu and Ding Hao and Bohai Zhao and Zhaoyang Liu and Zhiying Tu},
year = {2026},
eprint = {2608.03403},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2608.03403},
howpublished = {\url{https://github.com/WangCan1178/ExpG}}
}
```

98
benchmark/toolmemory/README_ZH.md Normal file
View file

@ -0,0 +1,98 @@
## Towards Robust Tool Use in Agents via Experience-Driven Adaptive Guidance
**语言**：中文 / [English](./README.md)
> 论文：[arXiv:2608.03403](https://arxiv.org/abs/2608.03403)
> 代码：[https://github.com/WangCan1178/ExpG](https://github.com/WangCan1178/ExpG)
<p align="center">
<img src="gitcha.png" alt="ExpG 挑战与概览" width="85%">
</p>
### 简介
本目录归档基于 [Agentscope ReMe](https://github.com/agentscope-ai/ReMe) 的工具使用增强工作 **ExpG**：在 ReMe 记忆框架之上，从历史工具调用中挖掘、提炼并复用经验，为智能体提供工具的 **能力边界** 与 **最佳实践指导**，从而：
- 在动态或有噪环境下更鲁棒地选择和调用工具；
- 让较小模型在带有经验指导时超越更大、但无记忆的基线；
- 在工具选择、工具调用和响应生成等多个阶段带来一致收益。
**如何使用 ReMe：** 启动 Tool Memory 服务后，历史工具调用经 `add_tool_call_result` 写入并评估，经 `summary_tool_memory` 蒸馏成工具级指导，再经 `retrieve_tool_memory` 取回并注入后续推理。向量存储与服务接口由 ReMe 提供，经验获取 / 蒸馏 / 复用策略由 ExpG 实现。完整实现与实验见 [WangCan1178/ExpG](https://github.com/WangCan1178/ExpG)。
---
### ExpG 机制概览
ExpG 将工具调用视为可学习经验，并通过三阶段流水线完成经验的获取、提炼与复用：
1. **经验获取（Experience Acquisition）**
- 从历史工具调用轨迹中分析调用质量（成功/失败、代价、时间等）；
- 针对不同工具构建结构化的经验单元，记录调用上下文、参数模式和结果。
2. **经验蒸馏（Experience Distillation）**
- 过滤无效 / 噪声经验，保留具有代表性的调用模式；
- 基于“等价类”视角对经验进行聚合，覆盖常见模式与稀有失败模式；
- 使用 LLM 对经验进行总结，形成可泛化的文本化指导（guidance）。
3. **经验复用（Experience Reuse）**
- 在未来任务中，根据当前工具调用上下文检索相关经验 / 指导；
- 将经验引导融入到工具选择、参数生成和响应整理等环节；
- 使得代理在面对动态环境和不完美反馈时仍能保持稳定表现。
---
### 主实验结果
MetaTool、API-Bank、BFCL-V3 上的性能对比（%）。**加粗**为各模型组内最优。
| Model | Method | MetaTool Pass@1 | MetaTool Avg@3 | MetaTool Pass@3 | API-Bank Pass@1 | API-Bank Avg@3 | API-Bank Pass@3 | BFCL-V3 Pass@1 | BFCL-V3 Avg@3 | BFCL-V3 Pass@3 | Total Pass@1 | Total Avg@3 | Total Pass@3 |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| GPT-5 nano | No Method | 72.62 | 72.76 | 78.49 | 82.96 | 83.46 | 86.97 | 53.80 | 53.00 | 60.95 | 70.82 | 70.62 | 76.63 |
| GPT-5 nano | Few-shot | 74.12 | 75.11 | 82.32 | 83.71 | 83.96 | **87.22** | 56.18 | 55.24 | 61.39 | 72.36 | 72.65 | 79.28 |
| GPT-5 nano | DRAFT | 73.94 | 73.04 | 78.97 | 84.21 | 83.46 | **87.22** | 57.27 | 57.27 | 62.26 | 72.52 | 71.58 | 77.23 |
| GPT-5 nano | Mem0 | 74.96 | 76.13 | 82.92 | 84.96 | 85.21 | **87.22** | 60.95 | 61.61 | 65.08 | 73.98 | 74.67 | 80.35 |
| GPT-5 nano | **ExpG** | **81.67** | **82.07** | **84.60** | **86.72** | **86.55** | **87.22** | **64.43** | **63.99** | **66.38** | **79.32** | **79.22** | **81.69** |
| DeepSeek-V3 | No Method | 83.10 | 82.94 | 84.66 | 84.71 | 84.38 | 85.46 | 58.79 | 59.65 | 65.94 | 78.92 | 78.66 | 81.37 |
| DeepSeek-V3 | Few-shot | 82.74 | 83.90 | 86.28 | 85.21 | 84.63 | 86.22 | 60.52 | 60.30 | 67.90 | 79.08 | 79.45 | 82.92 |
| DeepSeek-V3 | DRAFT | 80.23 | 80.79 | 82.44 | 84.96 | 85.63 | 86.47 | 62.26 | 61.61 | 68.55 | 77.70 | 77.80 | 80.54 |
| DeepSeek-V3 | Mem0 | 83.88 | 84.56 | 86.40 | 85.46 | 85.55 | 86.47 | 65.08 | 65.15 | 68.33 | 80.70 | 80.91 | 83.12 |
| DeepSeek-V3 | **ExpG** | **85.26** | **85.38** | **86.52** | **87.72** | **87.39** | **87.97** | **69.41** | **69.92** | **72.02** | **82.76** | **82.61** | **84.11** |
| Qwen3-8B | No Method | 76.51 | 76.97 | 77.71 | 83.96 | 83.88 | 84.21 | 58.79 | 58.28 | 60.30 | 74.46 | 74.41 | 75.56 |
| Qwen3-8B | Few-shot | 79.93 | 79.83 | 82.92 | 83.71 | 82.62 | 84.96 | 60.09 | 59.29 | 61.39 | 76.91 | 76.27 | 79.32 |
| Qwen3-8B | DRAFT | 78.19 | 77.33 | 77.89 | 85.71 | 84.96 | 85.46 | 60.74 | 60.30 | 62.91 | 76.20 | 75.18 | 76.35 |
| Qwen3-8B | Mem0 | 75.07 | 75.47 | 82.38 | 86.22 | 86.05 | 86.47 | 63.34 | 64.93 | 66.16 | 74.69 | 74.98 | 80.07 |
| Qwen3-8B | **ExpG** | **83.52** | **84.88** | **85.08** | **86.47** | **87.89** | **87.97** | **67.46** | **66.96** | **68.33** | **81.06** | **81.82** | **82.48** |
| Qwen3-32B | No Method | 80.05 | 79.43 | 80.17 | 84.71 | 84.88 | 85.21 | 65.15 | 65.08 | 66.16 | 78.05 | 77.55 | 78.41 |
| Qwen3-32B | **ExpG** | **84.68** | **85.02** | **86.28** | **86.97** | **87.30** | **87.72** | **70.72** | **71.01** | **73.32** | **82.48** | **82.56** | **84.14** |
| Qwen3-235B | No Method | 78.25 | 79.23 | 80.29 | 85.46 | 85.46 | 85.71 | 71.37 | 71.15 | 73.54 | 78.13 | 78.49 | 79.91 |
| Qwen3-235B | **ExpG** | **86.34** | **86.70** | **86.94** | **87.47** | **86.97** | **88.22** | **79.61** | **78.52** | **80.04** | **85.29** | **84.98** | **85.69** |
---
### 参考代码
| 路径 | 作用 |
| --- | --- |
| [`tool_memory.py`](./tool_memory.py) | 官方风格 ReMe Tool Memory HTTP 客户端（`add_tool_call_result` / `summary_tool_memory` / `retrieve_tool_memory`） |
| [`parse_tool_call_result_prompt.yaml`](./parse_tool_call_result_prompt.yaml) | 单次工具调用多维评估用的 prompt |
| [`summary_tool_memory_prompt.yaml`](./summary_tool_memory_prompt.yaml) | 将工具调用历史总结为 guidance 的 prompt |
| [`tool_memory_flows.yaml`](./tool_memory_flows.yaml) | Tool Memory 相关的 flow / op 配置摘录 |
以上为参考片段。完整可运行代码见 [WangCan1178/ExpG](https://github.com/WangCan1178/ExpG)。
---
### 引用
```bibtex
@misc{wang2026expg,
title = {Towards Robust Tool Use in Agents via Experience-Driven Adaptive Guidance},
author = {Can Wang and Haoran Chen and Li Yu and Ding Hao and Bohai Zhao and Zhaoyang Liu and Zhiying Tu},
year = {2026},
eprint = {2608.03403},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2608.03403},
howpublished = {\url{https://github.com/WangCan1178/ExpG}}
}
```

BIN
benchmark/toolmemory/gitcha.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 1.9 MiB

49
benchmark/toolmemory/parse_tool_call_result_prompt.yaml Normal file
View file

@ -0,0 +1,49 @@
prompt: |
You are an expert in evaluating tool invocation process. The tool is invoked by an AI agent.
Tool invocation Information:
- Tool Name: {tool_name}
- Success Flag: {success_flag}
- Time Cost: {time_cost}s
- Token Cost: {token_cost} tokens
- Agent Context: {context}
- Input Parameters: {input_params}
- Tool Response: {response}
- Tool Schema: {schema}
Evaluation Method:
Start from a default score list of scores = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0].
For each item below that is satisfied, assign 1 point to the corresponding index.
The final scores should be a list of 10 integers, each being either 0 or 1.
1. Use Quality (total 2 points. If context is provided, use it as an aid when evaluating):
- Index 1: Should the tool be invoked now? Consider whether all necessary information for the tool's invocation is ready, and whether the tool execution environment is correct. If it is a multi-round conversation, also consider the dependency relationships of the tool chain.
- Index 2: If should, is the chosen tool appropriate?
2. Input Quality (total 4 points. When evaluating, consider both the context and the tool schema):
- Index 3: Are all required parameters provided?
- Index 4: Are the input parameters valid and supported by the tool?
- Index 5: Are the input parameters in the correct format for their respective fields?
- Index 6: Does the value (content) of input parameter correctly reflect and match the given context?
3. Response Quality (total 4 points):
- Index 7: Does the response provide meaningful and useful information? Or are there any error messages or information that can be used as guidance for agent invoking tool better?
- Index 8: Does the response match the tool's intended purpose/function?
- Index 9: Does the response value correct (content appropriate) given the input parameters?
- Index 10: Does the response help accomplish the task within the given context?
Important:
1. Sometimes there is not enough information in the context or schema to make a complete evaluation. In such cases, make your best judgment based on the available information.
2. Some tools (commonly system tools such as mkdir, touch, echo, etc.) modify the external environment. Since these results cannot be obtained, they return "None" as the response. At this point, all the scores in the quality of the response should be obtained and should not be seen as a problem for the tool.
3. Evaluation independently from the success flag. The success_flag indicates whether the tool executed without technical errors. The evaluation should evaluate the quality of the tool invocation. A tool can execute successfully (Success Flag=1) but still produce low-quality or irrelevant responses, leading to a low evaluation score.
4. Sometimes an agent will execute multiple steps and invoke multiple tools to complete a task, but you only need to evaluate the use of one tool for one of the steps, not whether the final task is completed or not.
Answer Format:
Please provide your answer in the following JSON format:
```json
{
"scores": [0,0,0,0,0,0,0,0,0,0],
"explanation": "A brief evaluation (2-3 sentences) explaining the quality of the tool invocation, based on your evaluation. Low-quality aspects need to be reified, especially the causes of tool invocation errors."
}
```

32
benchmark/toolmemory/summary_tool_memory_prompt.yaml Normal file
View file

@ -0,0 +1,32 @@
prompt: |
You are an expert in analyzing tool usage patterns and generating practical usage guidance for agents.
Tool Information:
- Tool Name: {tool_name}
- Tool Schema: {tool_schema}
Recent Tool Invocation Experiences:
{experiences}
Important:
1. Assume the tool (tool schema) can't be changed, your task is to guide agent to use it better.
2. Your answer must be based on the information given, don't make it up. If not enough data, state "Not enough data to determine Core Function/Success Patterns/Common Issues/Best Practices."
3. Your answer will be used to guide the use of the tool in the future, so do not include content related to recent tool invocation experience such as "case #3" or "Call #2", but some values can be used as examples.
4. Pay attention to information not mentioned in the tool schema, such as the response upon successful tool invocation. It's also welcome to uncover insights, such as how tools can be used more effectively, and possible dependencies between tools. But if they aren't, don't make them up.
5. Finally, to avoid deriving incorrect guidance from individual invocation, check whether, if the agent follows the proposed guidance, it can perform better on all recent invocation histories. If not, revise the guidance until it can. Specifically:
- Don't write guidance in an absolute tone without a very deterministic message (meaning that all invocation histories are satisfied, otherwise it will result in failure).
- Sometimes there may be inconsistencies. Consider whether this is due to the context in which the tool is being used.
Your Task:
Based on the tool invocation history, generate a concise and logical tool usage guidance following this structure:
1. Core Function: What this tool does and when to use it.
2. Success Patterns: Parameter patterns and usage scenarios that work well.
3. Common Issues: Main pitfalls to avoid and why they fail.
4. Best Practices: 2-3 actionable recommendations.
Answer Format:
Provide a structured, concise guidance (max 200 words). Focus on actionable insights derived from actual usage data. Avoid generic advice and think step by step.
```txt
Your concise, data-driven tool usage guidance
```

234
benchmark/toolmemory/tool_memory.py Normal file
View file

@ -0,0 +1,234 @@
"""Official-style ReMe Tool Memory HTTP helpers.
Aligned with ReMe Tool Memory HTTP APIs (see ReMe cookbook
``use_tool_memory_demo.py`` and docs under ``docs/tool_memory/``):
- ``add_tool_call_result``
- ``summary_tool_memory``
- ``retrieve_tool_memory``
Response memories are read from ``metadata.memory_list[].content``.
This module does not use ExpG-only fields such as ``no_persist``,
``source_task``, or ``add_to``.
"""
from __future__ import annotations
import logging
from typing import Any, Dict, List, Optional
import httpx
logger = logging.getLogger(__name__)
DEFAULT_BASE_URL = "http://localhost:8002"
class ToolMemoryFetcher:
"""HTTP client for ReMe Tool Memory endpoints."""
def __init__(
self,
workspace_id: str,
base_url: str = DEFAULT_BASE_URL,
timeout: float = 60.0,
) -> None:
self.workspace_id = workspace_id
self.base_url = base_url.rstrip("/")
self.timeout = timeout
def _url(self, endpoint: str) -> str:
return f"{self.base_url}/{endpoint.lstrip('/')}"
@staticmethod
def _join_tool_names(tool_names: List[str] | str) -> str:
if isinstance(tool_names, str):
return tool_names
return ",".join(tool_names)
@staticmethod
def _memory_list(payload: Dict[str, Any]) -> List[Dict[str, Any]]:
metadata = payload.get("metadata") or {}
if not isinstance(metadata, dict):
return []
memory_list = metadata.get("memory_list") or []
return memory_list if isinstance(memory_list, list) else []
@classmethod
def _content_by_tool(cls, payload: Dict[str, Any]) -> Dict[str, str]:
result: Dict[str, str] = {}
for memory in cls._memory_list(payload):
if not isinstance(memory, dict):
continue
tool_name = str(memory.get("when_to_use") or "").strip()
content = memory.get("content") or ""
if tool_name:
result[tool_name] = str(content)
return result
async def add_tool_call_result_async(
self,
tool_call_results: List[Dict[str, Any]],
) -> Dict[str, Any]:
"""Call ``add_tool_call_result``."""
async with httpx.AsyncClient() as client:
response = await client.post(
self._url("add_tool_call_result"),
json={
"workspace_id": self.workspace_id,
"tool_call_results": tool_call_results,
},
timeout=self.timeout,
)
response.raise_for_status()
return response.json()
async def summary_tool_memory_async(
self,
tool_names: List[str] | str,
) -> Dict[str, Any]:
"""Call ``summary_tool_memory``."""
async with httpx.AsyncClient() as client:
response = await client.post(
self._url("summary_tool_memory"),
json={
"workspace_id": self.workspace_id,
"tool_names": self._join_tool_names(tool_names),
},
timeout=self.timeout,
)
response.raise_for_status()
return response.json()
async def retrieve_tool_memory_async(
self,
tool_names: List[str] | str,
) -> Dict[str, Any]:
"""Call ``retrieve_tool_memory``."""
async with httpx.AsyncClient() as client:
response = await client.post(
self._url("retrieve_tool_memory"),
json={
"workspace_id": self.workspace_id,
"tool_names": self._join_tool_names(tool_names),
},
timeout=self.timeout,
)
response.raise_for_status()
return response.json()
async def collect_memory_async(
self,
tool_names: List[str],
) -> Dict[str, str]:
"""Summarize then retrieve guidance for tools.
Returns:
Mapping from tool name to memory ``content`` string.
"""
if not tool_names:
return {}
names = self._join_tool_names(tool_names)
try:
summary = await self.summary_tool_memory_async(names)
if not summary.get("success"):
logger.warning("summary_tool_memory failed for %s", names)
except Exception as exc: # noqa: BLE001
logger.warning("summary_tool_memory error for %s: %s", names, exc)
try:
retrieved = await self.retrieve_tool_memory_async(names)
except Exception as exc: # noqa: BLE001
logger.warning("retrieve_tool_memory error for %s: %s", names, exc)
return {}
if not retrieved.get("success"):
logger.warning("retrieve_tool_memory failed for %s", names)
return {}
return self._content_by_tool(retrieved)
def add_tool_call_result(
self,
tool_call_results: List[Dict[str, Any]],
) -> Dict[str, Any]:
"""Sync wrapper for ``add_tool_call_result``."""
with httpx.Client() as client:
response = client.post(
self._url("add_tool_call_result"),
json={
"workspace_id": self.workspace_id,
"tool_call_results": tool_call_results,
},
timeout=self.timeout,
)
response.raise_for_status()
return response.json()
def summary_tool_memory(self, tool_names: List[str] | str) -> Dict[str, Any]:
"""Sync wrapper for ``summary_tool_memory``."""
with httpx.Client() as client:
response = client.post(
self._url("summary_tool_memory"),
json={
"workspace_id": self.workspace_id,
"tool_names": self._join_tool_names(tool_names),
},
timeout=self.timeout,
)
response.raise_for_status()
return response.json()
def retrieve_tool_memory(self, tool_names: List[str] | str) -> Dict[str, Any]:
"""Sync wrapper for ``retrieve_tool_memory``."""
with httpx.Client() as client:
response = client.post(
self._url("retrieve_tool_memory"),
json={
"workspace_id": self.workspace_id,
"tool_names": self._join_tool_names(tool_names),
},
timeout=self.timeout,
)
response.raise_for_status()
return response.json()
def collect_memory(self, tool_names: List[str]) -> Dict[str, str]:
"""Sync wrapper for summarize + retrieve.
Prefer ``collect_memory_async`` inside an existing event loop.
"""
if not tool_names:
return {}
names = self._join_tool_names(tool_names)
try:
summary = self.summary_tool_memory(names)
if not summary.get("success"):
logger.warning("summary_tool_memory failed for %s", names)
except Exception as exc: # noqa: BLE001
logger.warning("summary_tool_memory error for %s: %s", names, exc)
try:
retrieved = self.retrieve_tool_memory(names)
except Exception as exc: # noqa: BLE001
logger.warning("retrieve_tool_memory error for %s: %s", names, exc)
return {}
if not retrieved.get("success"):
logger.warning("retrieve_tool_memory failed for %s", names)
return {}
return self._content_by_tool(retrieved)
def get_memory_content(
self,
tool_names: List[str] | str,
) -> Optional[str]:
"""Retrieve and join memory contents for the given tools."""
payload = self.retrieve_tool_memory(tool_names)
if not payload.get("success"):
return None
contents = [content for content in self._content_by_tool(payload).values() if content]
return "\n\n".join(contents) if contents else None

45
benchmark/toolmemory/tool_memory_flows.yaml Normal file
View file

@ -0,0 +1,45 @@
# Tool Memory flow / op config excerpt used by ExpG.
# Full runnable code: https://github.com/WangCan1178/ExpG
flow:
retrieve_tool_memory:
flow_content: retrieve_tool_memory_op
description: "Retrieves tool memories from the vector database based on tool names to provide tool usage patterns and best practices"
input_schema:
tool_names:
type: string
description: "Comma-separated tool names (e.g., 'tool_name1,tool_name2')"
required: true
add_tool_call_result:
flow_content: parse_tool_call_result_op >> update_vector_store_op
description: "Evaluates and adds tool call results to the tool memory database, creating new memory or updating existing memory for the specified tool"
input_schema:
tool_call_results:
type: array
description: "List of tool call result objects, each containing: tool_name, input, output, success, time_cost, token_cost, create_time"
required: true
summary_tool_memory:
flow_content: summary_tool_memory_op >> update_vector_store_op
description: "Analyzes tool call history and generates comprehensive usage patterns, best practices, and recommendations for the specified tools"
input_schema:
tool_names:
type: string
description: "Comma-separated tool names to summarize (e.g., 'tool_name1,tool_name2')"
required: true
op:
parse_tool_call_result_op:
backend: parse_tool_call_result_op
llm: default
params:
max_history_tool_call_cnt: 100
evaluation_sleep_interval: 1.0
summary_tool_memory_op:
backend: summary_tool_memory_op
llm: default
params:
data_from: '2025-09-10 10:56:58'
summary_sleep_interval: 1.0

5
reme/components/runtime_context.py
View file

@ -73,6 +73,11 @@ class RuntimeContext:
await self._enqueue(StreamChunk(chunk_type=chunk_type, chunk=chunk))
return self
async def add_stream_chunk(self, chunk: StreamChunk) -> "RuntimeContext":
"""Emit a complete chunk while preserving session and tool metadata."""
await self._enqueue(chunk)
return self
async def add_stream_done(self) -> "RuntimeContext":
"""Emit the terminal DONE marker to close the stream."""
await self._enqueue(StreamChunk(chunk_type=ChunkEnum.DONE, chunk="", done=True))

42
reme/config/beam.yaml
View file

@ -164,6 +164,33 @@ jobs:
- messages
steps:
- backend: beam_auto_memory_step
# Long sessions are split into turn-aligned segments and fed to the
# agent incrementally; each segment holds at most this many words.
# ("segment" here is a slice of the message list, unrelated to file
# chunking in the index.) <= 0 disables splitting.
max_segment_words: 10000
# ── Text compression (direct LLM call, no agent) ──
compressor:
backend: base
description: "Compress text via a direct LLM call, optionally guided by queries as relevance filter"
parameters:
type: object
properties:
text:
type: string
description: "the text to compress"
queries:
type: array
description: "optional list of queries; content potentially relevant to any query is kept, content certainly irrelevant to all queries may be dropped"
items:
type: string
default: []
required:
- text
steps:
- backend: compressor_step
as_llm: compressor
# ── Reindex (full rebuild) ──
reindex:
@ -218,7 +245,7 @@ jobs:
- backend: search_v2_step
vector_weight: 0.7
candidate_multiplier: 5.0
expand_links: true
expand_links: false
max_links_per_direction: 10
add_draft:
@ -511,6 +538,19 @@ components:
parameters:
max_tokens: 65536
thinking_enable: true
compressor:
backend: ${LLM_BACKEND:-openai}
model: ${LLM_MODEL_NAME:-qwen3.6-flash}
stream: false
context_size: 200000
max_retries: 5
retry_delay: 5.0
credential:
api_key: ${LLM_API_KEY:-}
base_url: ${LLM_BASE_URL:-}
parameters:
max_tokens: 65536
thinking_enable: false
agent_wrapper:
default:

93
reme/config/default.yaml
View file

@ -211,6 +211,36 @@ jobs:
steps:
- backend: version_step
app_config:
backend: base
description: "return the effective application config with secrets redacted"
parameters:
type: object
properties: { }
steps:
- backend: app_config_step
chat:
backend: stream
description: "Stream a read-only agent conversation over the ReMe workspace."
parameters:
type: object
properties:
query:
type: string
description: "user message"
session_id:
type: string
description: "agent session to resume"
system_prompt:
type: string
description: "optional system prompt override"
required:
- query
steps:
- backend: chat_step
agent_wrapper: default
health_check:
backend: base
description: "return a concise health-check snapshot of reme components"
@ -258,19 +288,29 @@ jobs:
traverse:
backend: base
description: "Walk the wikilink graph from a path."
description: >-
Return a bounded wikilink graph with frontend-ready nodes and directed edges.
parameters:
type: object
properties:
path:
type: string
description: "path"
oneOf:
- type: string
- type: array
items:
type: string
minItems: 1
description: "workspace-relative path or paths used as graph seeds"
depth:
type: integer
description: "hop limit"
minimum: 0
description: "maximum number of wikilink hops"
default: 1
direction:
type: string
description: >-
traversal direction; returned edges always preserve the original
wikilink source-to-target direction
enum:
- forward
- backward
@ -281,6 +321,15 @@ jobs:
steps:
- backend: traverse_step
graph_snapshot:
backend: base
description: "Return the category-rooted digest wikilink graph with daily-note leaves."
parameters:
type: object
properties: { }
steps:
- backend: graph_snapshot_step
reindex:
backend: base
description: "wipe the file store and rebuild it from the existing files"
@ -523,6 +572,21 @@ jobs:
with_neighbors: false
max_neighbors_per_direction: 10
load:
backend: base
description: "Load a complete text file for the local editor without agent-output truncation."
parameters:
type: object
properties:
path:
type: string
description: "workspace-relative path"
required:
- path
steps:
- backend: load_step
max_bytes: 5242880
read_image:
backend: base
description: "Read an image file as base64 (workspace-relative path)."
@ -569,6 +633,27 @@ jobs:
steps:
- backend: write_step
save:
backend: base
description: "Save text verbatim, optionally rejecting external changes since the file was opened."
parameters:
type: object
properties:
path:
type: string
description: "workspace-relative path"
content:
type: string
description: "complete file content, including frontmatter"
expected_mtime:
type: string
description: "optional ISO mtime returned by stat"
required:
- path
- content
steps:
- backend: save_step
daily_write:
backend: base
description: "Write a daily markdown note with conversation source frontmatter."

37
reme/config/lme.yaml
View file

@ -163,6 +163,28 @@ jobs:
steps:
- backend: lme_auto_memory_step
# ── Text compression (direct LLM call, no agent) ──
compressor:
backend: base
description: "Compress text via a direct LLM call, optionally guided by queries as relevance filter"
parameters:
type: object
properties:
text:
type: string
description: "the text to compress"
queries:
type: array
description: "optional list of queries; content potentially relevant to any query is kept, content certainly irrelevant to all queries may be dropped"
items:
type: string
default: []
required:
- text
steps:
- backend: compressor_step
as_llm: compressor
# ── Reindex (full rebuild) ──
reindex:
backend: base
@ -216,7 +238,7 @@ jobs:
- backend: search_v2_step
vector_weight: 0.7
candidate_multiplier: 5.0
expand_links: true
expand_links: false
max_links_per_direction: 10
add_draft:
@ -508,6 +530,19 @@ components:
parameters:
max_tokens: 65536
thinking_enable: true
compressor:
backend: ${LLM_BACKEND:-openai}
model: ${LLM_MODEL_NAME:-qwen3.6-flash}
stream: false
context_size: 200000
max_retries: 5
retry_delay: 5.0
credential:
api_key: ${LLM_API_KEY:-}
base_url: ${LLM_BASE_URL:-}
parameters:
max_tokens: 65536
thinking_enable: false
agent_wrapper:
default:

8
reme/schema/__init__.py
View file

@ -35,12 +35,14 @@ from .dream import (
from .emb_node import EmbNode
from .file_chunk import FileChunk
from .file_front_matter import FileFrontMatter
from .graph_snapshot import GraphSnapshot, GraphSnapshotEdge, GraphSnapshotNode
from .file_link import FileLink
from .file_node import FileNode
from .request import Request
from .response import Response
from .stream_chunk import StreamChunk
from .token_usage import TokenUsage
from .traverse_graph import TraverseGraph, TraverseGraphEdge, TraverseGraphNode
__all__ = [
"ApplicationConfig",
@ -74,6 +76,9 @@ __all__ = [
"FileFrontMatter",
"FileLink",
"FileNode",
"GraphSnapshot",
"GraphSnapshotEdge",
"GraphSnapshotNode",
"IntegrateOutcome",
"JobConfig",
"PaperInfo",
@ -86,4 +91,7 @@ __all__ = [
"StreamChunk",
"TokenUsage",
"TopicSelectionOutput",
"TraverseGraph",
"TraverseGraphEdge",
"TraverseGraphNode",
]

32
reme/schema/graph_snapshot.py Normal file
View file

@ -0,0 +1,32 @@
"""Public response schemas for a complete indexed wikilink graph snapshot."""
from typing import Literal
from pydantic import BaseModel, Field
class GraphSnapshotNode(BaseModel):
"""One virtual category or indexed Markdown file in a graph snapshot."""
id: str = Field(description="Stable node identifier; documents use their workspace-relative path")
path: str = Field(description="Workspace-relative file or virtual category path")
name: str = Field(default="", description="Document name from frontmatter")
description: str = Field(default="", description="Document description from frontmatter")
indexed: bool = Field(description="Whether the target has an indexed FileNode")
virtual: bool = Field(default=False, description="Whether this is a generated category node")
class GraphSnapshotEdge(BaseModel):
"""One directed category or wikilink edge in a graph snapshot."""
source: str = Field(description="Source node identifier")
target: str = Field(description="Target node identifier")
target_anchor: str | None = Field(default=None, description="Optional heading, block, or line anchor")
class GraphSnapshot(BaseModel):
"""A category-rooted snapshot of digest wikilinks and their daily-note leaves."""
version: Literal[1] = 1
nodes: list[GraphSnapshotNode]
edges: list[GraphSnapshotEdge]

36
reme/schema/traverse_graph.py Normal file
View file

@ -0,0 +1,36 @@
"""Public response schemas for wikilink graph traversal."""
from typing import Literal
from pydantic import BaseModel, Field
class TraverseGraphNode(BaseModel):
"""One indexed file or unresolved wikilink target in a traversal result."""
id: str = Field(description="Stable node identifier; equal to the workspace-relative path")
path: str = Field(description="Workspace-relative file path")
name: str = Field(default="", description="Document name from frontmatter")
description: str = Field(default="", description="Document description from frontmatter")
depth: int = Field(ge=0, description="Shortest hop distance from any seed")
indexed: bool = Field(description="Whether the target has an indexed FileNode")
class TraverseGraphEdge(BaseModel):
"""One directed wikilink, preserving its original source and target."""
source: str = Field(description="Workspace-relative source file path")
target: str = Field(description="Workspace-relative target file path")
target_anchor: str | None = Field(default=None, description="Optional heading, block, or line anchor")
depth: int = Field(ge=1, description="Traversal depth at which this edge was first reached")
class TraverseGraph(BaseModel):
"""A bounded wikilink graph rooted at one or more workspace paths."""
version: Literal[1] = 1
seeds: list[str] = Field(description="Normalized workspace-relative traversal roots")
depth: int = Field(ge=0, description="Requested hop limit")
direction: Literal["forward", "backward", "both"]
nodes: list[TraverseGraphNode]
edges: list[TraverseGraphEdge]

31
reme/steps/benchmark/base/agentic_answer.py
View file

@ -13,6 +13,10 @@ class BaseAgenticAnswerStep(BaseStep):
Subclasses only need to set:
TOOL_CONTEXT_PREFIX (str): prefix used to build the unique tool_context_id.
JOB_TOOLS (list[str]): tools exposed to the ReAct agent; override to customize.
INJECTED_JOB_KWARGS (dict): server-owned kwargs injected into every job
tool call via ``injected_job_kwargs``; override the attribute or the
``_injected_job_kwargs`` hook to customize.
And apply their own ``@R.register(...)`` decorator and docstring.
@ -29,6 +33,29 @@ class BaseAgenticAnswerStep(BaseStep):
# backend's iteration-counting semantics; other backends ignore it.
MAX_ITERATION = 10
TOOL_CONTEXT_PREFIX: str = "content_agentic_answer"
JOB_TOOLS: list[str] = ["search", "add_draft", "read_all_draft", "read"]
INJECTED_JOB_KWARGS: dict = {"read_step_format_session": True}
def _injected_job_kwargs(self, query: str) -> dict: # pylint: disable=unused-argument
"""Server-owned kwargs injected into every job tool call.
Overridable hook: subclasses can extend the static
``INJECTED_JOB_KWARGS`` with per-request values derived from ``query``.
When the runtime context carries a truthy ``compress_session`` flag,
session-transcript compression is enabled in ``search_v2_step`` by
injecting a ``_search._compress.session`` marker plus the current
``query`` as the query-aware relevance filter. Default (falsy) leaves
session chunks uncompressed.
"""
injected = dict(self.INJECTED_JOB_KWARGS)
if self.context is not None and self.context.get("compress_session"):
injected["_search"] = {
"_compress": {"session": "true"},
"queries": [query],
"type": "query-aware",
}
return injected
async def execute(self):
assert self.context is not None
@ -54,10 +81,12 @@ class BaseAgenticAnswerStep(BaseStep):
tool_context_id = f"{self.TOOL_CONTEXT_PREFIX}_{os.getpid()}_local"
wrapper_kwargs = {
"system_prompt": sys_prompt,
"job_tools": ["search", "add_draft", "read_all_draft"],
"job_tools": list(self.JOB_TOOLS),
"react_config": {"max_iters": self.MAX_ITERATION},
"tool_context_id": tool_context_id,
}
if injected_job_kwargs := self._injected_job_kwargs(f"{query}(query time: {query_time})"):
wrapper_kwargs["injected_job_kwargs"] = injected_job_kwargs
if self.context.stream:
text = await self._stream_reply(query, **wrapper_kwargs)

6
reme/steps/benchmark/beam/agentic_answer.yaml
View file

@ -4,12 +4,16 @@ system_prompt: |
## Search Strategy
- You can call 'search' tool to search multiple times (at least once) with different queries to gather comprehensive information.
- Do not stop after the first relevant one unless you are confident that the information is sufficient.
- You can use 'read' tool to read specific parts of a specific file.
- After finding a preliminary answer, continue searching from multiple angles to check for potentially conflicting, supplementary, or updated information.
## Draft Tool
- Use 'add_draft' to save key findings during search, and 'read_all_draft' to review all saved notes before answering.
## Answer Rules
- Answer based ONLY on retrieved context.
- Output ONLY the direct factual answer — no reasoning, no search process, no elaboration.
- If information is not founded or not sufficient after multiple searches, reply: 'Information not found.'
- Do NOT invent or assume information that isn't in the memories. Do NOT extrapolate or fabricate connections from related but non-direct evidence.
- If information is not found or not sufficient after multiple searches, or if the maximum iteration count is reached without finding answer, reply: 'Information not found.'
temporal_hint: |
Current time context: {query_time}

158
reme/steps/benchmark/beam/auto_memory.py
View file

@ -5,8 +5,61 @@ from datetime import datetime, timedelta
from agentscope.message import Msg
from ...evolve.auto_memory import AutoMemoryStep, _normalize_msg_timestamp
from ...file_io import validate_session_id
from ....components import R
# Runtime-context key carrying the 0-based line offset of the current segment
# inside the full session file (segmented ingestion of long sessions).
_LINE_OFFSET_KEY = "beam_line_offset"
def _msg_word_count(msg: Msg) -> int:
"""Whitespace-separated word count of a message's text content."""
return len((msg.get_text_content() or "").split())
def split_turn_segments(messages: list[Msg], max_words: int) -> list[tuple[int, list[Msg]]]:
"""Split *messages* into segments of complete turns bounded by *max_words*.
A turn starts at every ``user`` message and spans all following non-user
messages (assistant replies). Segments only break at turn boundaries, so a
``user + assistant`` exchange is never split across two segments. A single
turn larger than *max_words* still becomes its own (oversized) segment.
Returns ``[(offset, msgs), ...]`` where ``offset`` is the 0-based index of
the segment's first message in *messages* — i.e. its line number in the
session file minus 1. ``max_words <= 0`` disables splitting.
"""
if not messages:
return []
if max_words <= 0:
return [(0, list(messages))]
# Group into turns: each user message opens a new turn.
turns: list[list[Msg]] = []
for msg in messages:
if msg.role == "user" or not turns:
turns.append([msg])
else:
turns[-1].append(msg)
segments: list[tuple[int, list[Msg]]] = []
current: list[Msg] = []
current_words = 0
offset = 0
for turn in turns:
turn_words = sum(_msg_word_count(m) for m in turn)
if current and current_words + turn_words > max_words:
segments.append((offset, current))
offset += len(current)
current = []
current_words = 0
current.extend(turn)
current_words += turn_words
if current:
segments.append((offset, current))
return segments
def _parse_iso_seconds(value: str) -> datetime | None:
"""Parse an ISO-8601 timestamp that is precise to at least seconds.
@ -121,8 +174,76 @@ class BeamAutoMemoryStep(AutoMemoryStep):
Date pinning for ``daily_write`` is handled by the base class through the
agent wrapper's server-owned ``injected_job_kwargs``.
Long sessions are ingested incrementally: the message list is split into
segments of complete turns (``max_segment_words`` step config, default
10000 words per segment) and each segment runs one agent pass — the first
pass creates the daily note, later passes merge into it. Line numbers
shown to the agent always refer to the ORIGINAL session file, not the
segment.
"""
async def execute(self):
assert self.context is not None
raw_messages = self.context.get("messages") or []
session_id: str = self.context.get("session_id", "")
max_words = int(self.kwargs.get("max_segment_words", 10000) or 0)
messages = self._build_messages(raw_messages)
segments = split_turn_segments(messages, max_words)
if len(segments) <= 1:
self.context[_LINE_OFFSET_KEY] = 0
await super().execute()
return
# Persist the FULL session file up-front so every segment's [Ln] labels
# (and the note's source markers) match the final on-disk layout. The
# per-segment saves inside the base execute become no-op appends.
if session_id and validate_session_id(session_id) is None:
await self._save_session_messages(session_id, messages)
base_hint = self.context.get("memory_hint", "") or ""
answers: list[str] = []
failed_segments = 0
self.logger.info(
f"[{self.name}] segmented ingestion session_id={session_id!r} "
f"messages={len(messages)} segments={len(segments)} max_segment_words={max_words}",
)
for part, (offset, segment) in enumerate(segments, start=1):
part_hint = (
f"This is part {part}/{len(segments)} of one long session "
f"(lines {offset + 1}-{offset + len(segment)} of the session file). "
"Earlier parts were already recorded; extract ONLY from the turns shown below."
)
self.context["messages"] = segment
self.context["memory_hint"] = f"{base_hint}\n{part_hint}".strip()
self.context[_LINE_OFFSET_KEY] = offset
# One bad segment (e.g. transiently corrupted note frontmatter) must
# not lose the remaining segments of the session.
try:
await super().execute()
ok = bool(self.context.response.success)
summary = self.context.response.answer or ""
except Exception as exc: # pylint: disable=broad-exception-caught
ok = False
summary = f"Error: {exc}"
self.logger.warning(
f"[{self.name}] segment {part}/{len(segments)} failed session_id={session_id!r}: {exc}",
)
if not ok:
failed_segments += 1
answers.append(f"[part {part}/{len(segments)}] {summary}")
# Restore caller-visible context and aggregate the response.
self.context["messages"] = raw_messages
self.context["memory_hint"] = base_hint
self.context.response.success = failed_segments < len(segments)
self.context.response.answer = "\n".join(answers)
self.context.response.metadata.update(
{"n_messages": len(messages), "segments": len(segments), "failed_segments": failed_segments},
)
def _build_messages(self, raw_messages: list) -> list[Msg]:
# Interpolate timestamps: if any message carries created_at, fill in
# the rest via linear interpolation so the whole session has coherent
@ -131,3 +252,40 @@ class BeamAutoMemoryStep(AutoMemoryStep):
[item if not isinstance(item, dict) else dict(item) for item in raw_messages],
)
return [self._to_msg(item) for item in interpolated]
def _format_history(self, messages: list[Msg]) -> str:
# Annotate every turn with its physical line number in the session
# file so the agent can cite information sources as
# [[<session file path>#L<start>-L<end>]]. A BEAM batch is already
# in chronological order, matching the one-message-per-line layout
# written by AutoMemoryStep._save_session_messages, so number each turn
# by its position plus the segment's offset into the full session file
# (no re-sorting). When the session is ingested in segments, the offset
# keeps [Ln] labels aligned with the ORIGINAL file, not the segment.
session_id = self.context.get("session_id", "") if self.context is not None else ""
line_offset = int(self.context.get(_LINE_OFFSET_KEY, 0) or 0) if self.context is not None else 0
source_path = self._session_source_path(session_id)
turns: list[str] = []
for line, msg in enumerate(messages, start=line_offset + 1):
text = (msg.get_text_content() or "").strip()
if not text:
continue
speaker = msg.name or msg.role or "?"
turns.append(f"[L{line} | {speaker} @ {msg.created_at}]\n{text}")
if not turns:
return "(empty)"
first_line = line_offset + 1
last_line = line_offset + len(messages)
preamble = (
f"Source file: {source_path}\n"
f"(Excerpt covering lines {first_line}-{last_line} of the source file. Each turn below is "
"prefixed with [Ln] = its line number in the source file; copy these numbers verbatim "
"into source markers.)"
)
return preamble + "\n\n" + "\n\n".join(turns)
def _reply_extra_kwargs(self, day: str) -> dict:
return {"tool_defaults": {"daily_write": {"date": day}}}

85
reme/steps/benchmark/beam/auto_memory.yaml
View file

@ -7,6 +7,13 @@ system_prompt: |
2. **Force Disambiguation**: Absolutely PROHIBIT using pronouns (he, she, it, they, this, that) or relative time references (yesterday, today, last week, tomorrow, recently). Always use full names, explicit dates, and specific references.
3. **Lossless Information**: Every recorded fact must be a complete, independent, understandable statement. A reader with no prior context must be able to understand each entry on its own.
4. **Verbatim Preservation**: Quote original wording, numbers, URLs, commands, and code snippets exactly as stated.
5. **Source Attribution**: Mark the source of factual content with a wikilink-style marker built from the `Source file` path (copied verbatim) and the `[Ln]` line numbers of the turns the fact was derived from. The EXACT format is `[[<path>#L<start>-L<end>(,L<start>-L<end>)*]]`, e.g. `[[session/dialog/beam_1M_1_batch1.jsonl#L1-L2]]`.
- **No `Source` label**: write the marker DIRECTLY at the end of the entry (or heading). Do NOT prefix it with `Source`, `Source:`, `source`, `来源`, or any other word — output only the bare `[[...]]` marker.
- **Double brackets + `#` + `L`-prefixed hyphen ranges**: EVERY marker MUST be wrapped in DOUBLE square brackets `[[...]]`, begin with the full session file path, then a hash `#`, then one or more line ranges. Each range is `L<start>-L<end>` — BOTH endpoints carry the `L` prefix and are joined by a hyphen `-` (NEVER a colon). A single line is written as a one-line range, e.g. `L41-L41`. This applies to EVERY comma-separated range too: `L69-L72,L77` is INVALID — write `L69-L72,L77-L77`. Multiple ranges of the same file go into ONE marker, separated by commas with NO spaces. ✗ WRONG: `[session/dialog/....jsonl:29-34]` (old single-bracket colon format), `[[...jsonl#29-34]]` (numbers missing `L`), `[[...jsonl#L29-34]]` (end missing `L`), `[[...jsonl#L29:L34]]` (colon instead of hyphen), `[[L29-L34]]` (path dropped), `[[...jsonl#L41]]` (bare single line — write `L41-L41`), `[[...jsonl#L69-L72,L77]]` (second range is a bare line — write `L77-L77`), `Source: [[...jsonl#L29-L34]]` (label added). ✓ RIGHT: `[[session/dialog/beam_1M_2_batch1.jsonl#L29-L34,L53-L58]]`.
- **Line numbers come from the `[Ln]` labels**: the `[Ln]` label prefixing each input turn (e.g. `[L29 | user @ ...]`) is that message's line number in the ORIGINAL source file — copy those numbers verbatim. The excerpt you see may START at a line far greater than 1; NEVER renumber from 1 and NEVER cite line numbers outside the excerpt's stated range. Newlines inside a message's own content do NOT affect its line number.
- **Merge same-file ranges into ONE marker**: instead of back-to-back markers like `[[...jsonl#L1-L2]][[...jsonl#L5-L6]]`, write a single marker with comma-separated ranges `[[...jsonl#L1-L2,L5-L6]]`.
- **Hoist shared markers, never repeat**: when several consecutive bullets share the SAME marker, do NOT append it to every bullet. Attach ONE marker to the group instead — either append it to the section heading (where it then covers every bullet beneath that heading until a different marker appears), or merge those bullets into a single entry carrying one trailing marker. Repeating an identical marker across sibling bullets is PROHIBITED — it wastes tokens.
- **Cite enough self-contained context**: the cited line range must be self-contained. Do NOT cite only the single line that states the key detail; widen the range to include the surrounding turns (e.g. the question or setup turn that gives the fact its meaning) so that a reader who opens only that line range understands the fact correctly and is not misled by missing context.
## What to Extract
@ -22,11 +29,11 @@ system_prompt: |
## Body Format
Use structured sections with headings. Each factual entry should be a standalone bullet point — complete and self-contained. Group entries by topic. Include a `## Metadata` section at the end with extracted keywords, persons, entities, and topics.
Use structured sections with headings. Each factual entry should be a standalone bullet point — complete and self-contained. Group entries by topic. Attach source markers coarsely (see the Source Attribution principle): prefer ONE marker per section heading or per grouped entry, write only the bare `[[...]]` marker with no `Source` label, and never repeat an identical marker across consecutive bullets. Do NOT add a `## Metadata` section.
## Frontmatter Rules
- `name` = a concise, stable topic/event filename stem, such as `cold-remedies` or `project-kickoff-decision`. Do not include today's date or the daily directory date; the outer daily path already records the date. For existing notes, update it when a better filename is clearly warranted.
- `name` = a concise, stable topic/event filename stem in kebab-case, such as `cold-remedies` or `project-kickoff-decision`. No date, no slash/backslash, none of `< > : " | ? *`. Do not include today's date or the daily directory date; the outer daily path already records the date. For existing notes, update it when a better filename is clearly warranted.
- `description` = a thorough, lossless summary; vague descriptions like "notes" / "misc" are unacceptable. The description alone should convey all key information.
- **Never set `status`** — it is a field reserved for downstream processing.
@ -37,6 +44,7 @@ system_prompt: |
- **NEVER** write tool invocations as text strings such as `daily_write name=...`. Text output does NOT execute tools — no memory will be saved.
- **ALWAYS** use the tool call mechanism provided by the runtime. The system handles parameter passing automatically.
- If you write a tool name and its parameters as text, NO tool will be executed and NO memory will be saved.
system_prompt_zh: |
你是自动记忆提取系统。你的任务是从对话中提取所有有价值的信息，并记录到结构化的日记记忆中。确保**完整覆盖**——每一条值得保留的信息都必须被捕获。
@ -46,6 +54,13 @@ system_prompt_zh: |
2. **强制消歧**：绝对禁止使用代词（他、她、它、他们、这个、那个）或相对时间（昨天、今天、上周、明天、最近）。必须使用全名、明确日期和具体引用。
3. **无损信息**：每条记录的事实必须是完整的、独立的、可理解的陈述。没有上下文的读者必须能单独理解每条记录。
4. **原文保留**：逐字引用原始措辞、数字、URL、命令和代码片段。
5. **来源标注**：为事实内容标注来源，使用 wikilink 风格的标记，由对话中显示的 `Source file` 路径（逐字复制）与该事实所依据对话轮次的 `[Ln]` 行号拼成。格式**必须**为 `[[<path>#L<start>-L<end>(,L<start>-L<end>)*]]`，例如 `[[session/dialog/beam_1M_1_batch1.jsonl#L1-L2]]`。
- **不要 `Source` 标签**：把标记**直接**写在条目（或标题）末尾。不要在前面加 `Source`、`Source:`、`source`、`来源` 或任何其他词——只输出裸的 `[[...]]` 标记。
- **双层方括号 + `#` + 带 `L` 前缀的连字符区间**：每个标记都**必须**用双层方括号 `[[...]]` 包裹，以完整的 session 文件路径开头，然后接 `#`，再接一个或多个行号区间。每个区间写作 `L<start>-L<end>`——两端都带 `L` 前缀，用连字符 `-` 连接（**绝不**用冒号）。单行引用写成单行区间，如 `L41-L41`；逗号分隔的每一个区间也同样适用：`L69-L72,L77` 是**无效的**，应写成 `L69-L72,L77-L77`。同一文件的多个区间放入**同一个**标记，用逗号分隔且不加空格。✗ 错误：`[session/dialog/....jsonl:29-34]`（旧的单括号冒号格式）、`[[...jsonl#29-34]]`（数字缺 `L`）、`[[...jsonl#L29-34]]`（末端缺 `L`）、`[[...jsonl#L29:L34]]`（用了冒号）、`[[L29-L34]]`（丢了路径）、`[[...jsonl#L41]]`（裸单行，应写 `L41-L41`）、`[[...jsonl#L69-L72,L77]]`（第二个区间是裸单行，应写 `L77-L77`）、`Source: [[...jsonl#L29-L34]]`（加了标签）。✓ 正确：`[[session/dialog/beam_1M_2_batch1.jsonl#L29-L34,L53-L58]]`。
- **行号来自 `[Ln]` 标签**：输入中每条对话轮次前的 `[Ln]` 标签（如 `[L29 | user @ ...]`）就是该消息在**原始**源文件中的行号——请逐字复制这些数字。你看到的片段可能从远大于 1 的行号开始；**绝不**要从 1 重新编号，也**绝不**要引用片段声明范围之外的行号。消息正文内部的换行不影响其行号。
- **同一文件多区间合并进一个标记**：不要写成相邻的多个标记 `[[...jsonl#L1-L2]][[...jsonl#L5-L6]]`，而要写成用逗号分隔区间的单个标记 `[[...jsonl#L1-L2,L5-L6]]`。
- **上提共享标记，绝不重复**：当连续多个条目共享**相同**标记时，不要给每个条目都追加。而是给整组只标注一个标记——要么把标记追加到分节标题上（此时它覆盖该标题下、直到出现不同标记之前的所有条目），要么把这些条目合并为一条并只在末尾带一个标记。在同级条目上重复相同标记是**被禁止的**——会浪费 token。
- **引用充分且自洽的上下文**：所标注的行号区间必须自洽完整。不要只标注陈述关键信息的那一句，要把区间扩大到包含周边轮次（例如赋予该事实意义的提问或铺垫轮次），使得仅打开该行号区间的读者也能正确理解这条事实，不会因缺失上下文而断章取义。
## 提取什么
@ -61,11 +76,11 @@ system_prompt_zh: |
## 正文格式
使用带标题的结构化分节。每条事实记录应是独立的列表项——完整且自包含。按主题分组。在末尾添加 `## Metadata` 部分，列出提取的关键词、人物、实体和主题。
使用带标题的结构化分节。每条事实记录应是独立的列表项——完整且自包含。按主题分组。粗粒度地标注来源（见来源标注原则）：优先每个分节标题或每组条目只用一个标记，只写裸的 `[[...]]` 标记且不带 `Source` 标签，并且绝不在连续条目上重复相同标记。不要添加 `## Metadata` 部分。
## Frontmatter 规则
- `name` = 简洁、稳定的主题/事件文件名 stem，例如 `cold-remedies` 或 `project-kickoff-decision`。不要包含今天日期或日记目录日期；外层日记路径已经记录日期。对已有笔记，如果明显有更好的文件名，就更新它。
- `name` = 简洁、稳定的主题/事件文件名 stem，采用 kebab-case，例如 `cold-remedies` 或 `project-kickoff-decision`。不含日期、不含 slash/反斜杠、不含 `< > : " | ? *`。不要包含今天日期或日记目录日期；外层日记路径已经记录日期。对已有笔记，如果明显有更好的文件名，就更新它。
- `description` = 详尽的无损总结；模糊的描述如 "notes" / "misc" 不可接受。仅凭 description 就应能传达全部核心信息。
- **永远不要设置 `status`**——它是下游处理保留的字段。
@ -81,6 +96,7 @@ user_message_create: |
Today: {today}
Extra hint: {note}
Session ID: {session_id}
Session file: {session_file} ← copy THIS exact path into every source marker (e.g. `[[{session_file}#L1-L2,L5-L6]]`)
# Recent Conversation
@ -115,15 +131,8 @@ user_message_create: |
⚠️ CRITICAL: You MUST invoke `daily_write` as a **tool call** (function call). Do NOT write `daily_write name=...` as plain text — that will NOT execute the tool and NO memory will be saved.
Body requirements:
- Each factual entry must be a complete, independent, self-contained statement (lossless restatement).
- PROHIBIT pronouns (he/she/it/they/this/that) — use full names and explicit references.
- PROHIBIT relative time (yesterday/today/last week) — use absolute dates (YYYY-MM-DD).
- Group entries by topic with headings.
- End with a `## Metadata` section containing: keywords, persons, entities, topic.
Filename and description rules:
- `name`: concise, stable topic/event filename stem in kebab-case. No date, no slash/backslash, no `< > : " | ? *`.
Body requirements — follow the system prompt's Extraction Principles, Body Format, and Frontmatter Rules (lossless entries; no pronouns; absolute dates; grouped by topic; no `## Metadata`). Two rules that MUST NOT be violated:
- Source markers: put a bare `[[session/dialog/....jsonl#L1-L2,L5-L6]]` at the END of an entry or heading — double brackets, full path + `#`, `L`-prefixed hyphen ranges, no `Source` label, coarse (hoist a shared marker, never repeat it across bullets), and cite a self-contained line range copied verbatim from the `[Ln]` labels.
- `description`: a thorough, lossless summary — specific enough that the description alone conveys all key information.
## Step 3 — Summary
@ -137,6 +146,7 @@ user_message_create_zh: |
今天：{today}
额外提示：{note}
Session ID：{session_id}
Session 文件：{session_file} ← 请把这个完整路径原样复制进每个来源标记（例如 `[[{session_file}#L1-L2,L5-L6]]`）
# 最近的对话
@ -171,15 +181,8 @@ user_message_create_zh: |
⚠️ 关键：你**必须**以**工具调用**（函数调用）方式调用 `daily_write`。不要将 `daily_write name=...` 作为纯文本输出——那样**不会**执行工具，也**不会**保存任何记忆。
正文要求：
- 每条事实记录必须是完整的、独立的、自包含的陈述（无损复述）。
- 禁止使用代词（他/她/它/他们/这个/那个）——必须使用全名和明确引用。
- 禁止使用相对时间（昨天/今天/上周）——必须使用绝对日期（YYYY-MM-DD）。
- 按主题分组并使用标题。
- 末尾添加 `## Metadata` 部分，包含：keywords、persons、entities、topic。
文件名和描述规则：
- `name`：简洁、稳定的主题/事件文件名 stem，采用 kebab-case。不含日期、不含 slash/反斜杠、不含 `< > : " | ? *`。
正文要求——遵循 system prompt 的提取原则、正文格式与 Frontmatter 规则（无损；不用代词；绝对日期；按主题分组；无 `## Metadata`）。以下两条**绝不能违反**：
- 来源标记：在条目或标题**末尾**放一个裸的 `[[session/dialog/....jsonl#L1-L2,L5-L6]]`——双层方括号、完整路径 + `#`、带 `L` 前缀的连字符区间、无 `Source` 标签、粒度粗（上提共享标记、绝不在连续条目上重复），且行号逐字复制自 `[Ln]` 标签、引用自洽完整的行号区间。
- `description`：详尽的无损总结——具体到仅凭 description 就能传达全部核心信息。
## 步骤 3 — 总结
@ -194,6 +197,7 @@ user_message_update: |
Today: {today}
Extra hint: {note}
Target path: {note_path}
Session file: {session_file} ← copy THIS exact path into every source marker (e.g. `[[{session_file}#L1-L2,L5-L6]]`)
# Recent Conversation
@ -219,19 +223,16 @@ user_message_update: |
The note already has content. Extract new information from the conversation and merge into it.
Extraction rules (apply to ALL new entries):
- Each new fact must be a complete, independent, self-contained statement (lossless restatement).
- PROHIBIT pronouns (he/she/it/they/this/that) — use full names and explicit references.
- PROHIBIT relative time (yesterday/today/last week) — use absolute dates (YYYY-MM-DD).
Extraction rules (apply to ALL new entries) — follow the system prompt's Extraction Principles and Body Format (lossless; no pronouns; absolute dates; grouped by topic). Source markers MUST be a bare `[[session/dialog/....jsonl#L1-L2,L5-L6]]` — double brackets, full path + `#`, `L`-prefixed hyphen ranges copied verbatim from the `[Ln]` labels, no `Source` label, coarse (never repeated across bullets), self-contained range.
Merge rules:
- **Timeline / history entries**: append only, never delete existing entries.
- **Current-state entries** (progress, blockers, next steps, open questions): rewrite the entire section to reflect the latest snapshot.
- **Everything else**: merge and deduplicate — keep all old facts, add new facts, remove exact duplicates.
- **Metadata section**: merge keywords, persons, entities, topics — union of old and new values.
Execution:
1. **Call the `edit` tool** for each section that needs updating, with parameters: `path`="{note_path}", `old`=<original fragment>, `new`=<replacement fragment>. You may call `edit` multiple times. Each call MUST be a tool call, not text.
- ⚠️ `edit` is for the note BODY only. NEVER touch the frontmatter block (the `---`-delimited YAML at the top of the file) with `edit` — a malformed YAML edit corrupts the whole note. All frontmatter changes MUST go through the `frontmatter_update` tool.
2. After body changes, **call the `frontmatter_update` tool** to refresh frontmatter, with parameters: `path`="{note_path}", `metadata`={{"name": "<updated filename stem>", "description": "<updated summary>"}}.
- Keep the existing `name` only when it is already the best concise topic/event filename stem. The system will rename the file after your final response.
- Do not add today's date or the daily directory date to `name`.
@ -241,16 +242,7 @@ user_message_update: |
The file exists but its body is empty. Extract and write the full content by **calling the `write` tool** with parameters: `path`="{note_path}", `name`=<name>, `description`=<description>, `content`=<body>.
Body requirements:
- Each factual entry must be a complete, independent, self-contained statement.
- PROHIBIT pronouns — use full names and explicit references.
- PROHIBIT relative time — use absolute dates (YYYY-MM-DD).
- Group entries by topic with headings.
- End with a `## Metadata` section containing: keywords, persons, entities, topic.
Filename and description rules:
- `name`: concise, stable topic/event filename stem in kebab-case. No date.
- `description`: a thorough, lossless summary — the description alone should convey all key information.
Body requirements — follow the system prompt's Extraction Principles, Body Format, and Frontmatter Rules (lossless; no pronouns; absolute dates; grouped by topic; no `## Metadata`). Source markers MUST be a bare `[[session/dialog/....jsonl#L1-L2,L5-L6]]` — double brackets, full path + `#`, `L`-prefixed hyphen ranges copied verbatim from the `[Ln]` labels, no `Source` label, coarse (never repeated across bullets), self-contained range. `description` must be a thorough, lossless summary that alone conveys all key information.
## Step 4 — Summary
@ -265,6 +257,7 @@ user_message_update_zh: |
今天：{today}
额外提示：{note}
目标路径：{note_path}
Session 文件：{session_file} ← 请把这个完整路径原样复制进每个来源标记（例如 `[[{session_file}#L1-L2,L5-L6]]`）
# 最近的对话
@ -290,19 +283,16 @@ user_message_update_zh: |
笔记已有内容。从对话中提取新信息并合并进去。
提取规则（适用于所有新条目）：
- 每条新事实必须是完整的、独立的、自包含的陈述（无损复述）。
- 禁止使用代词（他/她/它/他们/这个/那个）——必须使用全名和明确引用。
- 禁止使用相对时间（昨天/今天/上周）——必须使用绝对日期（YYYY-MM-DD）。
提取规则（适用于所有新条目）——遵循 system prompt 的提取原则与正文格式（无损；不用代词；绝对日期；按主题分组）。来源标记必须是裸的 `[[session/dialog/....jsonl#L1-L2,L5-L6]]`——双层方括号、完整路径 + `#`、带 `L` 前缀的连字符区间（行号逐字复制自 `[Ln]` 标签）、无 `Source` 标签、粒度粗（不在连续条目上重复）、自洽完整的行号区间。
合并规则：
- **时间线 / 历史条目**：仅追加，永远不删除已有条目。
- **当下状态类条目**（进度、卡点、下一步、未决问题）：整段重写，反映最新快照。
- **其余内容**：合并去重——保留全部旧事实，添加新事实，去除完全重复项。
- **Metadata 部分**：合并 keywords、persons、entities、topics——取旧值和新值的并集。
执行：
1. **调用 `edit` 工具**更新需要修改的部分，参数为：`path`="{note_path}"、`old`=<原文片段>、`new`=<替换片段>。可以多次调用 `edit`。每次调用必须是工具调用，不是文本。
- ⚠️ `edit` 只能用于笔记**正文**。**绝不**要用 `edit` 碰 frontmatter 块（文件顶部由 `---` 包围的 YAML）——格式错误的 YAML 会损坏整个笔记。所有 frontmatter 修改都必须通过 `frontmatter_update` 工具。
2. 正文变更后，**调用 `frontmatter_update` 工具**刷新 frontmatter，参数为：`path`="{note_path}"、`metadata`={{"name": "<更新后的文件名 stem>", "description": "<更新后的总结>"}}。
- 只有当前 `name` 已经是最合适的简洁主题/事件文件名 stem 时才保留。系统会在你最终回复后负责重命名文件。
- 不要在 `name` 中加入今天日期或日记目录日期。
@ -312,16 +302,7 @@ user_message_update_zh: |
文件存在但正文为空。提取并**调用 `write` 工具**一次性写入完整内容，参数为：`path`="{note_path}"、`name`=<name>、`description`=<description>、`content`=<正文>。
正文要求：
- 每条事实记录必须是完整的、独立的、自包含的陈述。
- 禁止使用代词——必须使用全名和明确引用。
- 禁止使用相对时间——必须使用绝对日期（YYYY-MM-DD）。
- 按主题分组并使用标题。
- 末尾添加 `## Metadata` 部分，包含：keywords、persons、entities、topic。
文件名和描述规则：
- `name`：简洁、稳定的主题/事件文件名 stem，采用 kebab-case。不含日期。
- `description`：详尽的无损总结——仅凭 description 就应能传达全部核心信息。
正文要求——遵循 system prompt 的提取原则、正文格式与 Frontmatter 规则（无损；不用代词；绝对日期；按主题分组；无 `## Metadata`）。来源标记必须是裸的 `[[session/dialog/....jsonl#L1-L2,L5-L6]]`——双层方括号、完整路径 + `#`、带 `L` 前缀的连字符区间（行号逐字复制自 `[Ln]` 标签）、无 `Source` 标签、粒度粗（不在连续条目上重复）、自洽完整的行号区间。`description` 必须是仅凭自身即可传达全部核心信息的详尽无损总结。
## 步骤 4 — 总结

2
reme/steps/benchmark/beam/llm_judge.py
View file

@ -14,7 +14,7 @@ A ``semantic`` alignment path is also available which uses ReMe's
configured ``as_embedding`` model (replacing BEAM's sentence_transformers).
This replicates the evaluation logic from
``benchmark/datasets/BEAM/src/evaluation/compute_metrics.py``.
``benchmark/beam/dataset/BEAM/src/evaluation/compute_metrics.py``.
"""
import json

4
reme/steps/benchmark/lme/agentic_answer.py
View file

@ -11,6 +11,10 @@ class LmeAgenticAnswerStep(BaseAgenticAnswerStep):
The agent uses the ``agent_wrapper`` component in ReAct mode, calling the
``search`` job tool to retrieve relevant memory chunks before generating
a final answer.
Session-transcript compression in ``search_v2_step`` is controlled by the
``compress_session`` flag in the runtime context (set by the benchmark
runner from ``evaluation.compress_session``); it is off by default.
"""
TOOL_CONTEXT_PREFIX = "lme_agentic_answer"

32
reme/steps/benchmark/lme/agentic_answer.yaml
View file

@ -1,18 +1,28 @@
system_prompt: |
You are a memory retrieval assistant. You MUST use the search tool to find information before answering.
- Your total time of tool calls should be at most 9 times
You are a memory retrieval assistant. A memory system manages the user's conversation history. Given a query, you MUST call the `search` tool at least once to retrieve relevant content from the memory system before answering.
## Memory System
The memory system stores the user's history in three forms:
- **Summary notes**: condensed knowledge distilled from past sessions.
- **Compressed session chunks**: fragments of session transcripts compressed to drop low-information phrasing; the dialogue structure is generally kept, but some detail may be shortened or lost.
- **Original sessions**: complete verbatim transcripts of past conversations, stored as memory files.
The `search` tool retrieves summary notes and compressed session chunks; it does NOT return original sessions directly. Every result is a chunk — a fragment of a memory file identified by its path — and may lack surrounding context. Compressed session chunks are marked with a leading `compressed session chunk:` notice. When a chunk looks highly relevant to the query but its content is vague, truncated, or missing detail due to compression, use the `read` tool with the chunk's path to read the original session before drawing conclusions.
## Workflow
1. Use `search` to locate possibly relevant content. It is a hybrid retrieval tool combining BM25 keyword matching and vector semantic search, ensuring diverse recall. Call it multiple times with different query phrasings to gather comprehensive information; do not stop after the first relevant result unless you are confident the information is sufficient.
2. When a retrieved chunk is strongly related to the query but ambiguous because of compression, call `read` on its path to recover the full original session, and ground your answer on that.
3. Use `add_draft` to save key findings during the search, and `read_all_draft` to review all saved notes before answering.
4. After finding a preliminary answer, continue searching from multiple angles to check for potentially conflicting, supplementary, or updated information. The goal is to improve the accuracy and completeness of the final answer.
5. Stop searching and generate the final answer once you are confident that further searches will not yield additional useful information.
- Your total tool calls should be at most 9 times.
## Search Strategy
- You can call 'search' tool to search multiple times (at least once) with different queries to gather comprehensive information.
- Do not stop after the first relevant one unless you are confident that the information is sufficient.
## Draft Tool
- Use 'add_draft' to save key findings during search, and 'read_all_draft' to review all saved notes before answering.
## Answer Rules
- Answer based ONLY on retrieved context.
- If the memories contain contradictory information, integrate the context, query, and most importantly the timestamp of each memory to comprehensively determine the final answer.
- Answer based ONLY on retrieved or read context.
- If memories contain contradictory information, weigh the context, query, and most importantly the timestamp of each memory to determine the final answer.
- Output ONLY the direct factual answer — no reasoning, no search process, no elaboration.
- Do NOT invent or assume information that isn't in the memories.
- If information is not founded or not sufficient after multiple searches, reply: 'Information not found.'
- Do NOT invent or assume information that isn't in the memories. Do NOT extrapolate or fabricate connections from related but non-direct evidence.
- If information is not found or not sufficient after multiple searches, or if the maximum iteration count is reached without finding answer, reply: 'Information not found.'
temporal_hint: |
Current time context: {query_time}

28
reme/steps/benchmark/lme/auto_memory.py
View file

@ -131,3 +131,31 @@ class LmeAutoMemoryStep(AutoMemoryStep):
[item if not isinstance(item, dict) else dict(item) for item in raw_messages],
)
return [self._to_msg(item) for item in interpolated]
def _format_history(self, messages: list[Msg]) -> str:
# Annotate every turn with its physical line number in the session
# file so the agent can cite information sources as
# [[<session file path>#L<start>-L<end>]]. A LongMemEval session is
# already in chronological order, matching the one-message-per-line
# layout written by AutoMemoryStep._save_session_messages, so number
# each turn by its position in the list (no re-sorting).
session_id = self.context.get("session_id", "") if self.context is not None else ""
source_path = self._session_source_path(session_id)
turns: list[str] = []
for line, msg in enumerate(messages, start=1):
text = (msg.get_text_content() or "").strip()
if not text:
continue
speaker = msg.name or msg.role or "?"
turns.append(f"[L{line} | {speaker} @ {msg.created_at}]\n{text}")
if not turns:
return "(empty)"
preamble = (
f"Source file: {source_path}\n"
f"(Each turn below is prefixed with [Ln] = its line number in the source file; "
"copy these numbers verbatim into source markers.)"
)
return preamble + "\n\n" + "\n\n".join(turns)

80
reme/steps/benchmark/lme/auto_memory.yaml
View file

@ -7,6 +7,13 @@ system_prompt: |
2. **Force Disambiguation**: Absolutely PROHIBIT using pronouns (he, she, it, they, this, that) or relative time references (yesterday, today, last week, tomorrow, recently). Always use full names, explicit dates, and specific references.
3. **Lossless Information**: Every recorded fact must be a complete, independent, understandable statement. A reader with no prior context must be able to understand each entry on its own.
4. **Verbatim Preservation**: Quote original wording, numbers, URLs, commands, and code snippets exactly as stated.
5. **Source Attribution**: Mark the source of factual content with a wikilink-style marker built from the `Source file` path (copied verbatim) and the `[Ln]` line numbers of the turns the fact was derived from. The EXACT format is `[[<path>#L<start>-L<end>(,L<start>-L<end>)*]]`, e.g. `[[session/dialog/s1.jsonl#L1-L2]]`.
- **No `Source` label**: write the marker DIRECTLY at the end of the entry (or heading). Do NOT prefix it with `Source`, `Source:`, `source`, `来源`, or any other word — output only the bare `[[...]]` marker.
- **Double brackets + `#` + `L`-prefixed hyphen ranges**: EVERY marker MUST be wrapped in DOUBLE square brackets `[[...]]`, begin with the full session file path, then a hash `#`, then one or more line ranges. Each range is `L<start>-L<end>` — BOTH endpoints carry the `L` prefix and are joined by a hyphen `-` (NEVER a colon). A single line is written as a one-line range, e.g. `L41-L41`. This applies to EVERY comma-separated range too: `L69-L72,L77` is INVALID — write `L69-L72,L77-L77`. Multiple ranges of the same file go into ONE marker, separated by commas with NO spaces. ✗ WRONG: `[session/dialog/....jsonl:29-34]` (old single-bracket colon format), `[[...jsonl#29-34]]` (numbers missing `L`), `[[...jsonl#L29-34]]` (end missing `L`), `[[...jsonl#L29:L34]]` (colon instead of hyphen), `[[L29-L34]]` (path dropped), `[[...jsonl#L41]]` (bare single line — write `L41-L41`), `[[...jsonl#L69-L72,L77]]` (second range is a bare line — write `L77-L77`), `Source: [[...jsonl#L29-L34]]` (label added). ✓ RIGHT: `[[session/dialog/s1.jsonl#L29-L34,L53-L58]]`.
- **Line numbers come from the `[Ln]` labels**: the `[Ln]` label prefixing each input turn (e.g. `[L29 | user @ ...]`) is that message's line number in the ORIGINAL source file — copy those numbers verbatim. Newlines inside a message's own content do NOT affect its line number.
- **Merge same-file ranges into ONE marker**: instead of back-to-back markers like `[[...jsonl#L1-L2]][[...jsonl#L5-L6]]`, write a single marker with comma-separated ranges `[[...jsonl#L1-L2,L5-L6]]`.
- **Hoist shared markers, never repeat**: when several consecutive bullets share the SAME marker, do NOT append it to every bullet. Attach ONE marker to the group instead — either append it to the section heading (where it then covers every bullet beneath that heading until a different marker appears), or merge those bullets into a single entry carrying one trailing marker. Repeating an identical marker across sibling bullets is PROHIBITED — it wastes tokens.
- **Cite enough self-contained context**: the cited line range must be self-contained. Do NOT cite only the single line that states the key detail; widen the range to include the surrounding turns (e.g. the question or setup turn that gives the fact its meaning) so that a reader who opens only that line range understands the fact correctly and is not misled by missing context.
## What to Extract
@ -22,7 +29,7 @@ system_prompt: |
## Body Format
Use structured sections with headings. Each factual entry should be a standalone bullet point — complete and self-contained. Group entries by topic. Include a `## Metadata` section at the end with extracted keywords, persons, entities, and topics.
Use structured sections with headings. Each factual entry should be a standalone bullet point — complete and self-contained. Group entries by topic. Attach source markers coarsely (see the Source Attribution principle): prefer ONE marker per section heading or per grouped entry, write only the bare `[[...]]` marker with no `Source` label, and never repeat an identical marker across consecutive bullets. Do NOT add a `## Metadata` section.
## Frontmatter Rules
@ -46,6 +53,13 @@ system_prompt_zh: |
2. **强制消歧**：绝对禁止使用代词（他、她、它、他们、这个、那个）或相对时间（昨天、今天、上周、明天、最近）。必须使用全名、明确日期和具体引用。
3. **无损信息**：每条记录的事实必须是完整的、独立的、可理解的陈述。没有上下文的读者必须能单独理解每条记录。
4. **原文保留**：逐字引用原始措辞、数字、URL、命令和代码片段。
5. **来源标注**：为事实内容标注来源，使用 wikilink 风格的标记，由对话中显示的 `Source file` 路径（逐字复制）与该事实所依据对话轮次的 `[Ln]` 行号拼成。格式**必须**为 `[[<path>#L<start>-L<end>(,L<start>-L<end>)*]]`，例如 `[[session/dialog/s1.jsonl#L1-L2]]`。
- **不要 `Source` 标签**：把标记**直接**写在条目（或标题）末尾。不要在前面加 `Source`、`Source:`、`source`、`来源` 或任何其他词——只输出裸的 `[[...]]` 标记。
- **双层方括号 + `#` + 带 `L` 前缀的连字符区间**：每个标记都**必须**用双层方括号 `[[...]]` 包裹，以完整的 session 文件路径开头，然后接 `#`，再接一个或多个行号区间。每个区间写作 `L<start>-L<end>`——两端都带 `L` 前缀，用连字符 `-` 连接（**绝不**用冒号）。单行引用写成单行区间，如 `L41-L41`；逗号分隔的每一个区间也同样适用：`L69-L72,L77` 是**无效的**，应写成 `L69-L72,L77-L77`。同一文件的多个区间放入**同一个**标记，用逗号分隔且不加空格。✗ 错误：`[session/dialog/....jsonl:29-34]`（旧的单括号冒号格式）、`[[...jsonl#29-34]]`（数字缺 `L`）、`[[...jsonl#L29-34]]`（末端缺 `L`）、`[[...jsonl#L29:L34]]`（用了冒号）、`[[L29-L34]]`（丢了路径）、`[[...jsonl#L41]]`（裸单行，应写 `L41-L41`）、`[[...jsonl#L69-L72,L77]]`（第二个区间是裸单行，应写 `L77-L77`）、`Source: [[...jsonl#L29-L34]]`（加了标签）。✓ 正确：`[[session/dialog/s1.jsonl#L29-L34,L53-L58]]`。
- **行号来自 `[Ln]` 标签**：输入中每条对话轮次前的 `[Ln]` 标签（如 `[L29 | user @ ...]`）就是该消息在**原始**源文件中的行号——请逐字复制这些数字。消息正文内部的换行不影响其行号。
- **同一文件多区间合并进一个标记**：不要写成相邻的多个标记 `[[...jsonl#L1-L2]][[...jsonl#L5-L6]]`，而要写成用逗号分隔区间的单个标记 `[[...jsonl#L1-L2,L5-L6]]`。
- **上提共享标记，绝不重复**：当连续多个条目共享**相同**标记时，不要给每个条目都追加。而是给整组只标注一个标记——要么把标记追加到分节标题上（此时它覆盖该标题下、直到出现不同标记之前的所有条目），要么把这些条目合并为一条并只在末尾带一个标记。在同级条目上重复相同标记是**被禁止的**——会浪费 token。
- **引用充分且自洽的上下文**：所标注的行号区间必须自洽完整。不要只标注陈述关键信息的那一句，要把区间扩大到包含周边轮次（例如赋予该事实意义的提问或铺垫轮次），使得仅打开该行号区间的读者也能正确理解这条事实，不会因缺失上下文而断章取义。
## 提取什么
@ -61,7 +75,7 @@ system_prompt_zh: |
## 正文格式
使用带标题的结构化分节。每条事实记录应是独立的列表项——完整且自包含。按主题分组。在末尾添加 `## Metadata` 部分，列出提取的关键词、人物、实体和主题。
使用带标题的结构化分节。每条事实记录应是独立的列表项——完整且自包含。按主题分组。粗粒度地标注来源（见来源标注原则）：优先每个分节标题或每组条目只用一个标记，只写裸的 `[[...]]` 标记且不带 `Source` 标签，并且绝不在连续条目上重复相同标记。不要添加 `## Metadata` 部分。
## Frontmatter 规则
@ -81,6 +95,7 @@ user_message_create: |
Today: {today}
Extra hint: {note}
Session ID: {session_id}
Session file: {session_file} ← copy THIS exact path into every source marker (e.g. `[[{session_file}#L1-L2,L5-L6]]`)
# Recent Conversation
@ -115,15 +130,8 @@ user_message_create: |
⚠️ CRITICAL: You MUST invoke `daily_write` as a **tool call** (function call). Do NOT write `daily_write name=...` as plain text — that will NOT execute the tool and NO memory will be saved.
Body requirements:
- Each factual entry must be a complete, independent, self-contained statement (lossless restatement).
- PROHIBIT pronouns (he/she/it/they/this/that) — use full names and explicit references.
- PROHIBIT relative time (yesterday/today/last week) — use absolute dates (YYYY-MM-DD).
- Group entries by topic with headings.
- End with a `## Metadata` section containing: keywords, persons, entities, topic.
Filename and description rules:
- `name`: concise, stable topic/event filename stem in kebab-case. No date, no slash/backslash, no `< > : " | ? *`.
Body requirements — follow the system prompt's Extraction Principles, Body Format, and Frontmatter Rules (lossless entries; no pronouns; absolute dates; grouped by topic; no `## Metadata`). Two rules that MUST NOT be violated:
- Source markers: put a bare `[[session/dialog/....jsonl#L1-L2,L5-L6]]` at the END of an entry or heading — double brackets, full path + `#`, `L`-prefixed hyphen ranges, no `Source` label, coarse (hoist a shared marker, never repeat it across bullets), and cite a self-contained line range copied verbatim from the `[Ln]` labels.
- `description`: a thorough, lossless summary — specific enough that the description alone conveys all key information.
## Step 3 — Summary
@ -137,6 +145,7 @@ user_message_create_zh: |
今天：{today}
额外提示：{note}
Session ID：{session_id}
Session 文件：{session_file} ← 请把这个完整路径原样复制进每个来源标记（例如 `[[{session_file}#L1-L2,L5-L6]]`）
# 最近的对话
@ -171,15 +180,8 @@ user_message_create_zh: |
⚠️ 关键：你**必须**以**工具调用**（函数调用）方式调用 `daily_write`。不要将 `daily_write name=...` 作为纯文本输出——那样**不会**执行工具，也**不会**保存任何记忆。
正文要求：
- 每条事实记录必须是完整的、独立的、自包含的陈述（无损复述）。
- 禁止使用代词（他/她/它/他们/这个/那个）——必须使用全名和明确引用。
- 禁止使用相对时间（昨天/今天/上周）——必须使用绝对日期（YYYY-MM-DD）。
- 按主题分组并使用标题。
- 末尾添加 `## Metadata` 部分，包含：keywords、persons、entities、topic。
文件名和描述规则：
- `name`：简洁、稳定的主题/事件文件名 stem，采用 kebab-case。不含日期、不含 slash/反斜杠、不含 `< > : " | ? *`。
正文要求——遵循 system prompt 的提取原则、正文格式与 Frontmatter 规则（无损；不用代词；绝对日期；按主题分组；无 `## Metadata`）。以下两条**绝不能违反**：
- 来源标记：在条目或标题**末尾**放一个裸的 `[[session/dialog/....jsonl#L1-L2,L5-L6]]`——双层方括号、完整路径 + `#`、带 `L` 前缀的连字符区间、无 `Source` 标签、粒度粗（上提共享标记、绝不在连续条目上重复），且行号逐字复制自 `[Ln]` 标签、引用自洽完整的行号区间。
- `description`：详尽的无损总结——具体到仅凭 description 就能传达全部核心信息。
## 步骤 3 — 总结
@ -194,6 +196,7 @@ user_message_update: |
Today: {today}
Extra hint: {note}
Target path: {note_path}
Session file: {session_file} ← copy THIS exact path into every source marker (e.g. `[[{session_file}#L1-L2,L5-L6]]`)
# Recent Conversation
@ -219,19 +222,16 @@ user_message_update: |
The note already has content. Extract new information from the conversation and merge into it.
Extraction rules (apply to ALL new entries):
- Each new fact must be a complete, independent, self-contained statement (lossless restatement).
- PROHIBIT pronouns (he/she/it/they/this/that) — use full names and explicit references.
- PROHIBIT relative time (yesterday/today/last week) — use absolute dates (YYYY-MM-DD).
Extraction rules (apply to ALL new entries) — follow the system prompt's Extraction Principles and Body Format (lossless; no pronouns; absolute dates; grouped by topic). Source markers MUST be a bare `[[session/dialog/....jsonl#L1-L2,L5-L6]]` — double brackets, full path + `#`, `L`-prefixed hyphen ranges copied verbatim from the `[Ln]` labels, no `Source` label, coarse (never repeated across bullets), self-contained range.
Merge rules:
- **Timeline / history entries**: append only, never delete existing entries.
- **Current-state entries** (progress, blockers, next steps, open questions): rewrite the entire section to reflect the latest snapshot.
- **Everything else**: merge and deduplicate — keep all old facts, add new facts, remove exact duplicates.
- **Metadata section**: merge keywords, persons, entities, topics — union of old and new values.
Execution:
1. **Call the `edit` tool** for each section that needs updating, with parameters: `path`="{note_path}", `old`=<original fragment>, `new`=<replacement fragment>. You may call `edit` multiple times. Each call MUST be a tool call, not text.
- ⚠️ `edit` is for the note BODY only. NEVER touch the frontmatter block (the `---`-delimited YAML at the top of the file) with `edit` — a malformed YAML edit corrupts the whole note. All frontmatter changes MUST go through the `frontmatter_update` tool.
2. After body changes, **call the `frontmatter_update` tool** to refresh frontmatter, with parameters: `path`="{note_path}", `metadata`={{"name": "<updated filename stem>", "description": "<updated summary>"}}.
- Keep the existing `name` only when it is already the best concise topic/event filename stem. The system will rename the file after your final response.
- Do not add today's date or the daily directory date to `name`.
@ -241,16 +241,7 @@ user_message_update: |
The file exists but its body is empty. Extract and write the full content by **calling the `write` tool** with parameters: `path`="{note_path}", `name`=<name>, `description`=<description>, `content`=<body>.
Body requirements:
- Each factual entry must be a complete, independent, self-contained statement.
- PROHIBIT pronouns — use full names and explicit references.
- PROHIBIT relative time — use absolute dates (YYYY-MM-DD).
- Group entries by topic with headings.
- End with a `## Metadata` section containing: keywords, persons, entities, topic.
Filename and description rules:
- `name`: concise, stable topic/event filename stem in kebab-case. No date.
- `description`: a thorough, lossless summary — the description alone should convey all key information.
Body requirements — follow the system prompt's Extraction Principles, Body Format, and Frontmatter Rules (lossless; no pronouns; absolute dates; grouped by topic; no `## Metadata`). Source markers MUST be a bare `[[session/dialog/....jsonl#L1-L2,L5-L6]]` — double brackets, full path + `#`, `L`-prefixed hyphen ranges copied verbatim from the `[Ln]` labels, no `Source` label, coarse (never repeated across bullets), self-contained range. `description` must be a thorough, lossless summary that alone conveys all key information.
## Step 4 — Summary
@ -265,6 +256,7 @@ user_message_update_zh: |
今天：{today}
额外提示：{note}
目标路径：{note_path}
Session 文件：{session_file} ← 请把这个完整路径原样复制进每个来源标记（例如 `[[{session_file}#L1-L2,L5-L6]]`）
# 最近的对话
@ -290,19 +282,16 @@ user_message_update_zh: |
笔记已有内容。从对话中提取新信息并合并进去。
提取规则（适用于所有新条目）：
- 每条新事实必须是完整的、独立的、自包含的陈述（无损复述）。
- 禁止使用代词（他/她/它/他们/这个/那个）——必须使用全名和明确引用。
- 禁止使用相对时间（昨天/今天/上周）——必须使用绝对日期（YYYY-MM-DD）。
提取规则（适用于所有新条目）——遵循 system prompt 的提取原则与正文格式（无损；不用代词；绝对日期；按主题分组）。来源标记必须是裸的 `[[session/dialog/....jsonl#L1-L2,L5-L6]]`——双层方括号、完整路径 + `#`、带 `L` 前缀的连字符区间（行号逐字复制自 `[Ln]` 标签）、无 `Source` 标签、粒度粗（不在连续条目上重复）、自洽完整的行号区间。
合并规则：
- **时间线 / 历史条目**：仅追加，永远不删除已有条目。
- **当下状态类条目**（进度、卡点、下一步、未决问题）：整段重写，反映最新快照。
- **其余内容**：合并去重——保留全部旧事实，添加新事实，去除完全重复项。
- **Metadata 部分**：合并 keywords、persons、entities、topics——取旧值和新值的并集。
执行：
1. **调用 `edit` 工具**更新需要修改的部分，参数为：`path`="{note_path}"、`old`=<原文片段>、`new`=<替换片段>。可以多次调用 `edit`。每次调用必须是工具调用，不是文本。
- ⚠️ `edit` 只能用于笔记**正文**。**绝不**要用 `edit` 碰 frontmatter 块（文件顶部由 `---` 包围的 YAML）——格式错误的 YAML 会损坏整个笔记。所有 frontmatter 修改都必须通过 `frontmatter_update` 工具。
2. 正文变更后，**调用 `frontmatter_update` 工具**刷新 frontmatter，参数为：`path`="{note_path}"、`metadata`={{"name": "<更新后的文件名 stem>", "description": "<更新后的总结>"}}。
- 只有当前 `name` 已经是最合适的简洁主题/事件文件名 stem 时才保留。系统会在你最终回复后负责重命名文件。
- 不要在 `name` 中加入今天日期或日记目录日期。
@ -312,16 +301,7 @@ user_message_update_zh: |
文件存在但正文为空。提取并**调用 `write` 工具**一次性写入完整内容，参数为：`path`="{note_path}"、`name`=<name>、`description`=<description>、`content`=<正文>。
正文要求：
- 每条事实记录必须是完整的、独立的、自包含的陈述。
- 禁止使用代词——必须使用全名和明确引用。
- 禁止使用相对时间——必须使用绝对日期（YYYY-MM-DD）。
- 按主题分组并使用标题。
- 末尾添加 `## Metadata` 部分，包含：keywords、persons、entities、topic。
文件名和描述规则：
- `name`：简洁、稳定的主题/事件文件名 stem，采用 kebab-case。不含日期。
- `description`：详尽的无损总结——仅凭 description 就应能传达全部核心信息。
正文要求——遵循 system prompt 的提取原则、正文格式与 Frontmatter 规则（无损；不用代词；绝对日期；按主题分组；无 `## Metadata`）。来源标记必须是裸的 `[[session/dialog/....jsonl#L1-L2,L5-L6]]`——双层方括号、完整路径 + `#`、带 `L` 前缀的连字符区间（行号逐字复制自 `[Ln]` 标签）、无 `Source` 标签、粒度粗（不在连续条目上重复）、自洽完整的行号区间。`description` 必须是仅凭自身即可传达全部核心信息的详尽无损总结。
## 步骤 4 — 总结

4
reme/steps/common/__init__.py
View file

@ -1,6 +1,8 @@
"""common steps"""
from .add import AddStep
from .app_config import AppConfigStep
from .chat import ChatStep
from .demo import DemoEchoStep1, DemoEchoStep2
from .health_check import HealthCheckStep
from .help import HelpStep
@ -14,6 +16,8 @@ from .version import VersionStep
__all__ = [
"AddStep",
"AppConfigStep",
"ChatStep",
"DemoEchoStep1",
"DemoEchoStep2",
"HealthCheckStep",

85
reme/steps/common/app_config.py Normal file
View file

@ -0,0 +1,85 @@
"""Return the effective application configuration."""
from typing import Any
from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit
from ..base_step import BaseStep
from ...components import R
_SECRET_KEYS = {
"api_key",
"app_secret",
"authorization",
"cookie",
"password",
"private_key",
"proxy_authorization",
"secret",
"set_cookie",
"token",
}
_SECRET_SUFFIXES = ("_access_key", "_api_key", "_password", "_private_key", "_secret", "_token")
def _is_secret_key(key: str) -> bool:
normalized = key.lower().replace("-", "_")
return normalized in _SECRET_KEYS or normalized.endswith(_SECRET_SUFFIXES)
def _redact_url(value: str) -> str:
"""Redact credentials embedded in an HTTP-style URL."""
try:
parsed = urlsplit(value)
if not parsed.scheme or not parsed.netloc:
return value
netloc = parsed.netloc
if parsed.password is not None:
host = parsed.hostname or ""
if ":" in host:
host = f"[{host}]"
if parsed.port is not None:
host = f"{host}:{parsed.port}"
netloc = f"{parsed.username or ''}:***@{host}"
query = urlencode(
[
(key, "***" if _is_secret_key(key) else item)
for key, item in parse_qsl(parsed.query, keep_blank_values=True)
],
)
return urlunsplit((parsed.scheme, netloc, parsed.path, query, parsed.fragment))
except ValueError:
return value
def _redact(value: Any) -> Any:
if isinstance(value, dict):
redacted = {}
for key, item in value.items():
key_string = str(key)
if _is_secret_key(key_string) or (
key_string.lower() == "credential" and not isinstance(item, (dict, list))
):
redacted[key] = "***"
else:
redacted[key] = _redact(item)
return redacted
if isinstance(value, list):
return [_redact(item) for item in value]
if isinstance(value, str):
return _redact_url(value)
return value
@R.register("app_config_step")
class AppConfigStep(BaseStep):
"""Return the effective config without startup environment or secrets."""
async def execute(self):
assert self.context is not None
assert self.app_context is not None
config = self.app_context.app_config.model_dump(mode="json", exclude={"environment"})
self.context.response.answer = _redact(config)
return self.context.response

45
reme/steps/common/chat.py Normal file
View file

@ -0,0 +1,45 @@
"""Workspace-aware streaming chat step for the ReMe web interface."""
from ..base_step import BaseStep
from ...components import R
from ...enumeration import ChunkEnum
@R.register("chat_step")
class ChatStep(BaseStep):
"""Stream a read-only agent conversation over the current workspace."""
DEFAULT_SYSTEM_PROMPT = """You are ReMe Agent, an assistant for a local-first memory workspace.
Use the available ReMe tools when workspace facts are needed. Cite workspace-relative file paths
when referring to notes. Never invent file contents, and do not claim to have changed files because
this chat intentionally provides read-only tools. Reply in the user's language."""
READ_ONLY_TOOLS = ["search", "list", "read", "stat", "traverse"]
async def execute(self):
assert self.context is not None
query = str(self.context.get("query") or "").strip()
if not query:
self.context.response.success = False
self.context.response.answer = "Skipped: empty query"
return self.context.response
if self.agent_wrapper is None:
raise RuntimeError("chat_step requires an agent_wrapper")
wrapper_kwargs = {
"system_prompt": self.context.get("system_prompt") or self.DEFAULT_SYSTEM_PROMPT,
"job_tools": self.READ_ONLY_TOOLS,
}
if session_id := self.context.get("session_id"):
wrapper_kwargs["resume"] = str(session_id)
parts: list[str] = []
async for chunk in self.agent_wrapper.reply_stream(query, **wrapper_kwargs):
await self.context.add_stream_chunk(chunk)
if chunk.chunk_type == ChunkEnum.CONTENT and isinstance(chunk.chunk, str):
parts.append(chunk.chunk)
if chunk.session_id:
self.context.response.metadata["session_id"] = chunk.session_id
self.context.response.success = True
self.context.response.answer = "".join(parts).strip()
return self.context.response

2
reme/steps/evolve/__init__.py
View file

@ -4,6 +4,7 @@ from ._evolve import now
from .auto_memory import AutoMemoryStep
from .auto_memory_cc import AutoMemoryCCStep
from .auto_resource import AutoResourceStep
from .compressor import CompressorStep
from .dream import DreamExtractStep, DreamFinishStep, DreamIntegrateStep, DreamTopicsStep, ProactiveStep
__all__ = [
@ -11,6 +12,7 @@ __all__ = [
"AutoMemoryStep",
"AutoMemoryCCStep",
"AutoResourceStep",
"CompressorStep",
"DreamExtractStep",
"DreamFinishStep",
"DreamIntegrateStep",

16
reme/steps/evolve/auto_memory.py
View file

@ -71,8 +71,11 @@ class AutoMemoryStep(BaseStep):
def _session_path(self, session_id: str) -> Path:
return self.file_store.workspace_path / self._session_dir() / "dialog" / f"{session_id}.jsonl"
def _session_source_path(self, session_id: str) -> str:
return f"{self._session_dir()}/dialog/{session_id}.jsonl"
def _session_link(self, session_id: str) -> str:
return f"[[{self._session_dir()}/dialog/{session_id}.jsonl]]"
return f"[[{self._session_source_path(session_id)}]]"
def _daily_note_path(self, day: str, name: str) -> str:
return f"{self.config_value('daily_dir')}/{day}/{name}.md"
@ -240,6 +243,14 @@ class AutoMemoryStep(BaseStep):
"""
return {}
def _format_history(self, messages: list[Msg]) -> str:
"""Render the conversation history injected into the prompt.
Overridable hook: subclasses can annotate messages (e.g. with their
source line numbers in the session file) before rendering.
"""
return format_history(messages)
# pylint: disable=too-many-return-statements
async def execute(self):
assert self.context is not None
@ -308,7 +319,8 @@ class AutoMemoryStep(BaseStep):
note=memory_hint or "(none)",
note_path=note_path,
session_id=session_id,
history=format_history(messages),
session_file=self._session_source_path(session_id),
history=self._format_history(messages),
)
self.logger.info(f"[{self.name}] agent start path={note_path} template={template_key}")

82
reme/steps/evolve/compressor.py Normal file
View file

@ -0,0 +1,82 @@
"""compressor — compress text via a direct LLM call (no agent)."""
from agentscope.message import UserMsg
from agentscope.model import ChatResponse
from ..base_step import BaseStep
from ...components import R
@R.register("compressor_step")
class CompressorStep(BaseStep):
"""Compress ``text`` with a direct LLM call (no agent wrapper).
Inputs (from RuntimeContext):
text (str, required): the text to compress.
queries (list[str], optional, default []): when provided, compression
keeps anything potentially relevant to any of the queries and may
drop content that is certainly irrelevant and certainly unhelpful
for all of them.
Output (written to context.response.answer):
The compressed text.
"""
async def execute(self):
assert self.context is not None
text: str = self.context.get("text", "") or ""
raw_queries = self.context.get("queries") or []
queries: list[str] = [str(q).strip() for q in raw_queries if str(q).strip()]
if not text.strip():
self.context.response.success = False
self.context.response.answer = "Skipped: empty text"
return
if queries:
queries_block = "\n".join(f"- {q}" for q in queries)
user_message = self.prompt_format("compress_query_prompt", queries=queries_block, text=text)
else:
user_message = self.prompt_format("compress_prompt", text=text)
result = await self.as_llm([UserMsg(name="user", content=user_message)])
compressed = await self._response_text(result)
self.logger.info(
f"[{self.name}] compressed {len(text)} -> {len(compressed)} chars (queries={len(queries)})",
)
self.context.response.success = True
self.context.response.answer = compressed
self.context.response.metadata.update(
{
"queries": queries,
"original_length": len(text),
"compressed_length": len(compressed),
},
)
@staticmethod
async def _response_text(result) -> str:
"""Extract text blocks from a streaming or non-streaming ChatResponse.
The async-iteration check is done on the type: ``ChatResponse`` inherits
agentscope's ``DictMixin`` whose instance ``__getattr__`` raises
``KeyError`` (not ``AttributeError``), so ``hasattr(result, ...)``
would crash instead of returning False.
"""
if hasattr(type(result), "__aiter__"):
last: ChatResponse | None = None
async for chunk in result:
last = chunk
result = last
if result is None:
return ""
# Content blocks may be plain dicts or pydantic models (TextBlock/ThinkingBlock).
parts: list[str] = []
for block in result.content or []:
if isinstance(block, dict):
if block.get("type") == "text":
parts.append(str(block.get("text") or ""))
elif getattr(block, "type", None) == "text":
parts.append(str(getattr(block, "text", "") or ""))
return "".join(parts).strip()

91
reme/steps/evolve/compressor.yaml Normal file
View file

@ -0,0 +1,91 @@
compress_prompt: |
You are a text compression engine. The text below is a conversation transcript. Compress it so it becomes shorter while losing zero information.
## Compression Rules
1. Delete sentences that carry no real information: greetings, pleasantries, small talk, thanks, apologies, courtesy formulas, filler words, and empty transitions (e.g. "Hi there!", "Hope you're doing well", "Thanks so much!", "As we all know").
2. You may rewrite, merge, or restructure sentences to make them shorter, but you must NOT lose any information. Every fact, entity, number, date, condition, causal relation, opinion, preference, and nuance present in the original must remain fully recoverable from your output.
3. Never delete informative content, and never weaken it into a vaguer statement. Compression must come only from removing zero-information phrasing and from tighter wording.
4. Never add new information, comments, guesses, or interpretations of your own.
5. Copy critical details verbatim: numbers, dates, times, amounts, identifiers, proper names, quotes, file paths, and code must appear exactly as in the original. Vague or relative time expressions (e.g. "last Wednesday", "two weeks ago") must also be preserved as-is.
6. Keep the original language of the text, and try to preserve the original format (such as the `[role @ timestamp]` message lines) and the original order of information.
7. If the entire text carries no real information (pure greetings/small talk), output nothing.
## Output Format
Output ONLY the compressed text. No explanations, no preamble, no quotes, no markdown fences.
# Text
{text}
compress_prompt_zh: |
你是一个文本压缩引擎。下面的文本是一段对话记录。请压缩它，使其变短，但信息量零损失。
## 压缩规则
1. 删除没有实际信息量的语句：问候、寒暄、客套、感谢、道歉、口头禅、空洞的过渡语（例如"你好呀！""希望你一切都好""非常感谢！""众所周知"）。
2. 可以改写、合并、调整语句以减小长度，但不得损失任何信息。原文中的每一个事实、实体、数字、日期、条件、因果关系、观点、偏好和细微含义，都必须能从你的输出中完整还原。
3. 不得删减信息内容，也不得把信息弱化成更模糊的表述。压缩只能来自删除零信息量的表达和更紧凑的措辞。
4. 不得添加任何新信息、评论、猜测或你自己的解读。
5. 关键细节必须逐字保留：数字、日期、时间、金额、标识符、专有名词、引语、文件路径和代码必须与原文完全一致。模糊或相对的时间描述（如"上周三""两周前"）也必须原样保留。
6. 保持文本的原始语言，尽量保持原有的格式（如 `[role @ 时间戳]` 的消息行）和信息的原有顺序。
7. 如果整段文本没有任何实际信息（纯问候/寒暄），则输出空。
## 输出格式
只输出压缩后的文本。不要解释、不要开场白、不要引号、不要 markdown 代码块。
# 文本
{text}
compress_query_prompt: |
You are a text compression engine. The text below is a conversation transcript. Compress it so it becomes shorter, guided by the queries as a relevance filter, without losing any potentially useful information.
## Compression Rules
1. Delete sentences that carry no real information: greetings, pleasantries, small talk, thanks, apologies, courtesy formulas, filler words, and empty transitions (e.g. "Hi there!", "Hope you're doing well", "Thanks so much!", "As we all know").
2. You may rewrite, merge, or restructure sentences to make them shorter, but the kept content must NOT lose information. Every fact, entity, number, date, condition, causal relation, opinion, preference, and nuance you keep must remain fully recoverable from your output.
3. Keep ALL content that is possibly or potentially related to ANY of the queries. Even weak, indirect, background, or uncertain relevance to any single query means it MUST be kept.
4. You may delete a piece of information ONLY when both conditions hold for EVERY query: it is certainly unrelated to that query, AND it is certainly of no help whatsoever for answering that query.
5. If you are unsure whether something can be deleted, KEEP it. When in doubt, always err on the side of keeping.
6. Do NOT answer the queries. Do NOT add new information, comments, guesses, or interpretations of your own.
7. Copy critical details verbatim: numbers, dates, times, amounts, identifiers, proper names, quotes, file paths, and code must appear exactly as in the original. Vague or relative time expressions (e.g. "last Wednesday", "two weeks ago") must also be preserved as-is.
8. Keep the original language of the text, and try to preserve the original format (such as the `[role @ timestamp]` message lines) and the original order of information.
## Output Format
Output ONLY the compressed text. No explanations, no preamble, no quotes, no markdown fences.
# Queries
{queries}
# Text
{text}
compress_query_prompt_zh: |
你是一个文本压缩引擎。下面的文本是一段对话记录。请以下列 query 列表作为相关性过滤条件，压缩文本使其变短，同时不丢失任何可能有用的信息。
## 压缩规则
1. 删除没有实际信息量的语句：问候、寒暄、客套、感谢、道歉、口头禅、空洞的过渡语（例如"你好呀！""希望你一切都好""非常感谢！""众所周知"）。
2. 可以改写、合并、调整语句以减小长度，但保留下来的内容不得损失信息。你保留的每一个事实、实体、数字、日期、条件、因果关系、观点、偏好和细微含义，都必须能从你的输出中完整还原。
3. 保留任何可能、潜在与任意一个 query 相关的内容。哪怕只与其中某一个 query 相关性微弱、间接、只是背景信息或不确定，也必须保留。
4. 只有对每一个 query 都同时满足以下两个条件时，才可以删除一条信息：它一定与该 query 无关，并且对回答该 query 一定没有任何帮助。
5. 对于不确定是否可以删除的信息，一律保留。拿不准时，永远倾向于保留。
6. 不要回答这些 query 本身。不得添加任何新信息、评论、猜测或你自己的解读。
7. 关键细节必须逐字保留：数字、日期、时间、金额、标识符、专有名词、引语、文件路径和代码必须与原文完全一致。模糊或相对的时间描述（如"上周三""两周前"）也必须原样保留。
8. 保持文本的原始语言，尽量保持原有的格式（如 `[role @ 时间戳]` 的消息行）和信息的原有顺序。
## 输出格式
只输出压缩后的文本。不要解释、不要开场白、不要引号、不要 markdown 代码块。
# Query 列表
{queries}
# 文本
{text}

4
reme/steps/file_io/__init__.py
View file

@ -12,9 +12,11 @@ from .frontmatter_delete import FrontmatterDeleteStep
from .frontmatter_read import FrontmatterReadStep
from .frontmatter_update import FrontmatterUpdateStep
from .list import ListStep
from .load import LoadStep
from .move import MoveStep
from .read import ReadStep
from .read_image import ReadImageStep
from .save import SaveStep
from .stat import StatStep
from .write import WriteStep
@ -35,9 +37,11 @@ __all__ = [
"FrontmatterReadStep",
"FrontmatterUpdateStep",
"ListStep",
"LoadStep",
"MoveStep",
"ReadStep",
"ReadImageStep",
"SaveStep",
"StatStep",
"WriteStep",
]

42
reme/steps/file_io/_file_io.py
View file

@ -8,6 +8,7 @@ import aiofiles
import aiofiles.os
from ...constants import DEFAULT_MAX_BYTES, MAX_FILE_READ_BYTES, TRUNCATION_NOTICE_MARKER
from ...schema import FileChunk
from ...utils import get_logger
logger = get_logger(log_to_file=False)
@ -214,3 +215,44 @@ def truncate_text_output(
except Exception:
logger.warning("truncate_text_output failed, returning original text", exc_info=True)
return text
def truncate_session_output(
text: str,
*,
start_line: int = 1,
total_lines: int = 0,
max_bytes: int = DEFAULT_MAX_BYTES,
file_path: str | None = None,
encoding: str = "utf-8",
) -> str:
"""Render session jsonl lines then truncate by bytes preserving line integrity.
Each line of *text* is expected to be a serialized ``Msg`` JSON from a raw
session transcript (``*.jsonl``). Lines are first rendered via
:func:`~reme.steps.index._source_format.render_session_chunk_lines` — one
message per line as ``[speaker @ created_at] content`` with internal
newlines flattened — and then truncated with the same byte-budget logic
as :func:`truncate_text_output`.
"""
if not text or max_bytes <= 0:
return text
# Render each jsonl line into a human-readable [speaker @ time] content line.
# Import locally to avoid a hard dependency from file_io on the index package
# at module load time; the import is safe (no circular dependency).
from ..index._source_format import render_session_chunk_lines
chunk = FileChunk(text=text)
rendered_lines = render_session_chunk_lines(chunk)
rendered_text = "\n".join(rendered_lines)
# Reuse the same byte-budget truncation as truncate_text_output.
return truncate_text_output(
rendered_text,
start_line=start_line,
total_lines=total_lines,
max_bytes=max_bytes,
file_path=file_path,
encoding=encoding,
)

2
reme/steps/file_io/list.py
View file

@ -68,7 +68,7 @@ class ListStep(BaseStep):
out: list[str] = []
for entry in files:
try:
out.append(str(entry.relative_to(workspace_dir)))
out.append(entry.relative_to(workspace_dir).as_posix())
except ValueError:
out.append(str(entry))
return out

66
reme/steps/file_io/load.py Normal file
View file

@ -0,0 +1,66 @@
"""Load a complete text file for an interactive editor."""
from datetime import datetime
from ._file_io import read_file_safe
from ._path import _check_path_permission, resolve_path
from ..base_step import BaseStep
from ...components import R
DEFAULT_EDITOR_MAX_BYTES = 5 * 1024 * 1024
@R.register("load_step")
class LoadStep(BaseStep):
"""Return complete text without the truncation used by agent-facing reads."""
def _fail(self, message: str, **metadata) -> None:
assert self.context is not None
self.context.response.success = False
self.context.response.answer = f"Error: {message}"
self.context.response.metadata.update(metadata)
async def execute(self):
assert self.context is not None
raw_path = str(self.context.get("path") or "")
max_bytes = int(self.context.get("max_bytes") or self.kwargs.get("max_bytes") or DEFAULT_EDITOR_MAX_BYTES)
target, error = resolve_path(self.workspace_path, raw_path)
if error or target is None:
self._fail(error or "invalid path", path=raw_path)
return self.context.response
if not _check_path_permission(self.workspace_path, target, self.context.get("_allowed_paths")):
self._fail("no permission to read this file", path=raw_path)
return self.context.response
if not target.is_file():
self._fail("file does not exist", path=raw_path)
return self.context.response
stat = target.stat()
if stat.st_size > max_bytes:
self._fail(
f"file is too large for the editor ({stat.st_size} bytes; limit {max_bytes})",
code="file_too_large",
path=raw_path,
size=stat.st_size,
max_bytes=max_bytes,
)
return self.context.response
try:
content, encoding = await read_file_safe(target, max_bytes=max_bytes)
except Exception as exc: # pylint: disable=broad-except
self._fail(f"load failed: {exc}", path=raw_path)
return self.context.response
self.context.response.success = True
self.context.response.answer = content
self.context.response.metadata.update(
{
"path": raw_path,
"exists": True,
"type": "file",
"size": stat.st_size,
"mtime": datetime.fromtimestamp(stat.st_mtime).isoformat(),
"encoding": encoding,
},
)
return self.context.response

46
reme/steps/file_io/read.py
View file

@ -2,9 +2,10 @@
from pathlib import Path
from ._file_io import read_file_lines_safe, read_file_safe, truncate_text_output
from ._file_io import read_file_lines_safe, read_file_safe, truncate_session_output, truncate_text_output
from ._path import _check_path_permission, NON_MD_WARNING, gate_md, resolve_path
from ..base_step import BaseStep
from ..index._source_format import is_session_path
from ...components import R
from ...constants import DEFAULT_MAX_BYTES, MAX_FILE_READ_BYTES
from ...utils import expand_links, render_expansion_lines
@ -21,6 +22,14 @@ class ReadStep(BaseStep):
neighbors (out/in link targets) with name/description meta,
fetched via the file_store. Same rendering as SearchStep.
max_neighbors_per_direction (int, default 10): cap per direction.
read_step_format_session (bool, default False): when true and the
resolved path is a raw session transcript (``*.jsonl`` under the
dialog dir), render each jsonl line as
``[speaker @ created_at] content`` before truncation via
:func:`truncate_session_output`. Non-session files fall back to
the standard :func:`truncate_text_output` path. Injected at
runtime via ``injected_job_kwargs`` (takes precedence) or set in
the YAML ``steps`` config as a fallback.
Permission: honors the request-scoped ``_allowed_paths`` constraint
injected by the server into the RuntimeContext.
@ -31,6 +40,7 @@ class ReadStep(BaseStep):
assert self.context is not None
self.context.response.success = False
self.context.response.answer = f"Error: {message}"
self.logger.warning(f"[{self.name}] FAILED: {message} | meta={meta}")
if meta:
self.context.response.metadata.update(meta)
@ -92,6 +102,29 @@ class ReadStep(BaseStep):
self._fail(f"read failed: {e}", path=str(target))
return None
def _truncate_with_session_format(self, excerpt: str, target: Path, start_line: int, total: int) -> str:
"""Truncate *excerpt* using session-aware rendering when *target* is a session file.
Falls back to :func:`truncate_text_output` when the path is not under
the workspace or is not a raw session transcript.
"""
try:
rel_path = str(target.relative_to(self.workspace_path))
except ValueError:
self.logger.info(f"[{self.name}] skip session format: path outside workspace_path path={target}")
return truncate_text_output(excerpt, start_line=start_line, total_lines=total, file_path=str(target))
dialog_dir = self.config_value("dialog_dir")
if not is_session_path(rel_path, dialog_dir):
return truncate_text_output(excerpt, start_line=start_line, total_lines=total, file_path=str(target))
return truncate_session_output(
excerpt,
start_line=start_line,
total_lines=total,
file_path=str(target),
)
# pylint: disable=too-many-return-statements
async def execute(self):
assert self.context is not None
@ -99,6 +132,12 @@ class ReadStep(BaseStep):
start_line, end_line = self.context.get("start_line"), self.context.get("end_line")
with_neighbors: bool = bool(self.kwargs.get("with_neighbors", False))
max_neighbors_per_direction: int = int(self.kwargs.get("max_neighbors_per_direction", 10))
# Injected value takes precedence over YAML kwargs; check existence
# (not truthiness) so an explicit False can disable a YAML-true flag.
_format_session = self.context.get("read_step_format_session")
if _format_session is None:
_format_session = self.kwargs.get("read_step_format_session", False)
format_session: bool = bool(_format_session)
# Validate inputs and target before touching the filesystem twice.
target = self._resolve_target(raw)
@ -151,7 +190,10 @@ class ReadStep(BaseStep):
return None
e = min(total, requested_end if requested_end is not None else total)
text = truncate_text_output(excerpt, start_line=s, total_lines=total, file_path=str(target))
if format_session:
text = self._truncate_with_session_format(excerpt, target, s, total)
else:
text = truncate_text_output(excerpt, start_line=s, total_lines=total, file_path=str(target))
self.context.response.success = True
self.context.response.answer = text

78
reme/steps/file_io/save.py Normal file
View file

@ -0,0 +1,78 @@
"""Save a text file verbatim with optional optimistic-concurrency protection."""
from datetime import datetime
from ._file_io import detect_file_encoding, get_path_lock, write_file_safe
from ._path import _check_path_permission, resolve_path
from ..base_step import BaseStep
from ...components import R
@R.register("save_step")
class SaveStep(BaseStep):
"""Persist editor content without interpreting or rebuilding frontmatter.
``expected_mtime`` is the ISO timestamp returned by ``stat``. When supplied,
saving fails if the file changed after it was opened. This keeps the
filesystem authoritative when ReMe and another editor are used together.
"""
def _fail(self, message: str, **metadata) -> None:
assert self.context is not None
self.context.response.success = False
self.context.response.answer = f"Error: {message}"
self.context.response.metadata.update(metadata)
async def execute(self):
assert self.context is not None
raw_path = str(self.context.get("path") or "")
content = str(self.context.get("content") or "")
expected_mtime = self.context.get("expected_mtime")
target, error = resolve_path(self.workspace_path, raw_path)
if error or target is None:
self._fail(error or "invalid path", path=raw_path)
return self.context.response
if not _check_path_permission(self.workspace_path, target, self.context.get("_allowed_paths")):
self._fail("no permission to write this file", path=raw_path)
return self.context.response
if target.exists() and not target.is_file():
self._fail("target is not a file", path=raw_path)
return self.context.response
lock = await get_path_lock(target)
async with lock:
current_mtime = datetime.fromtimestamp(target.stat().st_mtime).isoformat() if target.exists() else None
if expected_mtime and current_mtime != str(expected_mtime):
self._fail(
"file changed outside ReMe; reload it before saving",
code="file_conflict",
path=raw_path,
expected_mtime=str(expected_mtime),
current_mtime=current_mtime,
)
return self.context.response
encoding = await detect_file_encoding(target) if target.exists() else "utf-8"
try:
await write_file_safe(target, content, encoding=encoding)
except Exception as exc: # pylint: disable=broad-except
self._fail(f"save failed: {exc}", path=raw_path)
return self.context.response
stat = target.stat()
saved_mtime = datetime.fromtimestamp(stat.st_mtime).isoformat()
self.context.response.success = True
self.context.response.answer = f"Saved {raw_path} ({stat.st_size} bytes)"
self.context.response.metadata.update(
{
"path": raw_path,
"exists": True,
"type": "file",
"size": stat.st_size,
"mtime": saved_mtime,
},
)
self.logger.info(f"[{self.name}] saved path={target} bytes={stat.st_size} encoding={encoding}")
return self.context.response

2
reme/steps/index/__init__.py
View file

@ -4,6 +4,7 @@ from .bm25_search import Bm25SearchStep
from .clear_paths import ClearPathsStep
from .clear_store import ClearStoreStep
from .draft import AddDraftStep, ReadAllDraftStep
from .graph_snapshot import GraphSnapshotStep
from .log_changes import LogChangesStep
from .node_search import NodeSearchStep
from .init_changes import InitChangesStep
@ -30,6 +31,7 @@ __all__ = [
"DEFAULT_LOW_POWER_POLL_MS",
"DEFAULT_WATCH_DEBOUNCE_MS",
"DEFAULT_WATCH_STEP_MS",
"GraphSnapshotStep",
"InitChangesStep",
"LogChangesStep",
"NodeSearchStep",

158
reme/steps/index/_source_format.py
View file

@ -1,16 +1,20 @@
"""Shared helpers: render retrieved chunks and assemble search-step answers.
Raw session transcripts (``*.jsonl`` under the dialog dir) store one serialized
``Msg`` per line. :func:`render_chunk_body` turns those back into a readable
dialog; all other chunks keep their raw ``text``. Used by
``Msg`` per line. :func:`render_chunk_body` renders those line-aligned — one
message per line with internal newlines flattened; all other chunks keep their
raw ``text``. Used by
``search``/``vector_search``/``bm25_search`` so every step renders session hits
identically.
:func:`format_chunks_answer` assembles a full answer string from a list of
chunks, with optional per-chunk score formatting and link expansion. Raw
session chunks from the same file whose line ranges overlap, contain one
another, or are adjacent are merged into their union before rendering (see
:func:`_merge_session_chunk_intervals`) so a passage is never shown twice.
:func:`render_chunk_entries` renders each chunk into a ``{"path",
"start_line", "end_line", "score", "body", "link"}`` entry dict, with optional
per-chunk score formatting and link expansion; :func:`join_chunk_entries`
assembles the source headers and joins those entries into the final answer
string. Chunks are rendered as given; callers that want raw session chunks from
the same file with overlapping, contained, or adjacent line ranges collapsed
into their union should pre-merge them via :func:`merge_session_chunk_intervals`
so a passage is never shown twice.
:data:`ALL_RETURNED_MESSAGE` is the English notice shown when tool_context dedup
removes every previously-returned result. :data:`NO_RESULTS_MESSAGE` is the
English notice shown when the search returned no results at all.
@ -20,7 +24,6 @@ from typing import Callable, Final
from agentscope.message import Msg
from ..evolve._evolve import format_history
from ...schema import FileChunk
from ...utils.link_expansion import render_expansion_lines
@ -35,36 +38,59 @@ ALL_RETURNED_MESSAGE: Final[str] = (
NO_RESULTS_MESSAGE: Final[str] = "No relevant information was found for the given query."
def is_session_chunk(chunk: FileChunk, dialog_dir: str) -> bool:
"""True if the chunk comes from a raw session transcript (a jsonl file under the dialog dir)."""
path = (chunk.path or "").strip().strip("/")
def is_session_path(path: str, dialog_dir: str) -> bool:
"""True if the path points at a raw session transcript (a jsonl file under the dialog dir)."""
path = (path or "").strip().strip("/")
if not path.endswith(".jsonl"):
return False
dialog_dir = (dialog_dir or "").strip("/")
return path == dialog_dir or path.startswith(f"{dialog_dir}/")
def render_chunk_body(chunk: FileChunk, dialog_dir: str) -> str:
"""Render a chunk's body, compacting raw session transcripts into a readable form.
def is_session_chunk(chunk: FileChunk, dialog_dir: str) -> bool:
"""True if the chunk comes from a raw session transcript (a jsonl file under the dialog dir)."""
return is_session_path(chunk.path, dialog_dir)
Session chunks are jsonl where each line is a serialized ``Msg``. Parse every line
and render via :func:`format_history`; on any parse error (or no usable messages),
fall back to the chunk's raw ``text``.
def render_chunk_body(chunk: FileChunk, dialog_dir: str) -> str:
"""Render a chunk's body; raw session transcripts render one message per line.
Session chunks are jsonl where each line is a serialized ``Msg``. They
render via :func:`render_session_chunk_lines` — one message per line with
internal newlines flattened — so verbatim and compressed session bodies
share the same single-line-per-message format. All other chunks keep
their raw ``text``.
"""
if not is_session_chunk(chunk, dialog_dir):
return chunk.text
try:
messages: list[Msg] = []
for line in chunk.text.splitlines():
line = line.strip()
if not line:
continue
messages.append(Msg.model_validate_json(line))
if not messages:
return chunk.text
return format_history(messages)
except Exception:
return chunk.text
return "\n".join(render_session_chunk_lines(chunk))
def render_session_chunk_lines(chunk: FileChunk) -> list[str]:
"""Render a raw session chunk line-aligned with the original jsonl file.
Output line ``i`` maps to file line ``chunk.start_line + i`` (1-based).
Each jsonl line (one serialized ``Msg`` per line) becomes a single rendered
line ``[speaker @ created_at] content`` with internal whitespace and
newlines flattened to single spaces. Blank lines stay blank and
unparseable lines pass through stripped, so the file-line mapping is
never broken.
"""
rendered: list[str] = []
for raw in chunk.text.splitlines():
line = raw.strip()
if not line:
rendered.append("")
continue
try:
msg = Msg.model_validate_json(line)
except Exception:
rendered.append(line)
continue
speaker = msg.name or msg.role or "?"
content = " ".join((msg.get_text_content() or "").split())
rendered.append(f"[{speaker} @ {msg.created_at}] {content}".rstrip())
return rendered
def _build_union_chunk(group: list[FileChunk]) -> FileChunk:
@ -97,7 +123,7 @@ def _build_union_chunk(group: list[FileChunk]) -> FileChunk:
)
def _merge_session_chunk_intervals(chunks: list[FileChunk], dialog_dir: str) -> list[FileChunk]:
def merge_session_chunk_intervals(chunks: list[FileChunk], dialog_dir: str) -> list[FileChunk]:
"""Merge raw session chunks from the same file into their line-range union.
Only chunks recognized as raw session transcripts (see
@ -155,39 +181,71 @@ def _finalize_group(group: list[FileChunk]) -> FileChunk:
return _build_union_chunk(group)
def format_chunks_answer(
def render_chunk_entries(
chunks: list[FileChunk],
dialog_dir: str,
*,
include_source: bool = True,
score_fn: Callable[[FileChunk], str] | None = None,
link_expansion: dict[str, dict] | None = None,
) -> str:
"""Render a list of chunks into a single answer string.
) -> list[dict[str, str]]:
"""Render each chunk into an entry dict of source fields, body, and link.
Raw session chunks from the same file that overlap, contain one another, or
are adjacent are merged into their union first (see
:func:`_merge_session_chunk_intervals`).
Chunks are rendered exactly as given, in input order. Callers that want
overlapping/contained/adjacent raw session chunks collapsed into their
union should pre-merge the list via :func:`merge_session_chunk_intervals`
before calling.
When *include_source* is ``True`` (default), each chunk is prefixed with a
source header showing path, line range, and score. When ``False``, only the
rendered bodies are included, separated by blank lines.
When *include_source* is ``True`` (default), each entry carries the source
fields ``path`` / ``start_line`` / ``end_line`` / ``score`` (the formatted
score string) plus ``body`` and ``link`` (the per-path expansion lines,
empty string when there is none). When ``False``, each entry carries only
``body``.
*score_fn* customizes the score string in the header (default
``"score={chunk.score:.4f}"``). *link_expansion* appends per-path expansion
lines after each chunk's header block (used by hybrid search).
*score_fn* customizes the formatted ``score`` string (default
``"score={chunk.score:.4f}"``). *link_expansion* provides the per-path
expansion lines (used by hybrid search).
"""
chunks = _merge_session_chunk_intervals(chunks, dialog_dir)
if not include_source:
return "\n\n".join(render_chunk_body(c, dialog_dir) for c in chunks)
return [{"body": render_chunk_body(c, dialog_dir)} for c in chunks]
fmt = score_fn or (lambda c: f"score={c.score:.4f}")
lines: list[str] = []
entries: list[dict[str, str]] = []
for c in chunks:
lines.append(
f"========== {c.path}:{c.start_line}-{c.end_line} "
f"[{fmt(c)}] ==========\n{render_chunk_body(c, dialog_dir)}",
entries.append(
{
"path": c.path,
"start_line": str(c.start_line),
"end_line": str(c.end_line),
"score": fmt(c),
"body": render_chunk_body(c, dialog_dir),
"link": "\n".join(render_expansion_lines((link_expansion or {}).get(c.path, {}))),
},
)
if link_expansion:
lines.extend(render_expansion_lines(link_expansion.get(c.path, {})))
return "\n".join(lines)
return entries
def join_chunk_entries(entries: list[dict[str, str]]) -> str:
"""Assemble entries into the final answer string.
For each entry, a source header line is rebuilt from ``path`` /
``start_line`` / ``end_line`` / ``score`` when ``path`` is present, then
followed by the non-empty ``body`` and ``link`` parts, joined with ``\\n``.
Entries are separated by a blank line; missing or empty parts are skipped.
"""
parts: list[str] = []
for entry in entries:
tmp = []
if path := entry.get("path", "").strip():
tmp.append(
f"========== {path}:{entry.get('start_line', '')}-{entry.get('end_line', '')} "
f"[{entry.get('score', '')}] ==========",
)
for key in ("body", "link"):
value = entry.get(key, "").strip()
if not value:
continue
tmp.append(value)
if tmp:
parts.append("\n".join(tmp))
return "\n\n".join(parts)

8
reme/steps/index/bm25_search.py
View file

@ -3,7 +3,8 @@
from typing import Final
from ._dedup import _ToolContextDedupMixin
from ._source_format import ALL_RETURNED_MESSAGE, NO_RESULTS_MESSAGE, format_chunks_answer
from ._source_format import ALL_RETURNED_MESSAGE, NO_RESULTS_MESSAGE, join_chunk_entries
from ._source_format import merge_session_chunk_intervals, render_chunk_entries
from ..base_step import BaseStep
from ...components import R
@ -48,11 +49,12 @@ class Bm25SearchStep(_ToolContextDedupMixin, BaseStep):
results = results[:limit]
dialog_dir = self.config_value("dialog_dir")
self.context.response.answer = format_chunks_answer(
results,
entries = render_chunk_entries(
merge_session_chunk_intervals(results, dialog_dir),
dialog_dir,
include_source=self.include_source,
)
self.context.response.answer = join_chunk_entries(entries)
if not results:
self.context.response.answer = ALL_RETURNED_MESSAGE if pre_dedup_count > 0 else NO_RESULTS_MESSAGE
self.context.response.metadata["results"] = [

86
reme/steps/index/graph_snapshot.py Normal file
View file

@ -0,0 +1,86 @@
"""Return the digest graph, rooted by its three memory categories."""
from ..base_step import BaseStep
from ...components import R
from ...enumeration import DreamBucketEnum
from ...schema import GraphSnapshot, GraphSnapshotEdge, GraphSnapshotNode
_CATEGORY_BUCKETS = (
DreamBucketEnum.WIKI,
DreamBucketEnum.PERSONAL,
DreamBucketEnum.PROCEDURE,
)
@R.register("graph_snapshot_step")
class GraphSnapshotStep(BaseStep):
"""Build the frontend digest graph through the file-store contract.
Category nodes connect to every indexed Markdown file in their bucket.
Digest files retain wikilinks to other digest files and to daily notes. Daily
notes are leaves: their own outgoing links are intentionally not returned.
"""
async def execute(self):
assert self.context is not None
indexed_nodes = await self.file_store.get_nodes()
node_by_path = {node.path: node for node in indexed_nodes if node.path.lower().endswith(".md")}
digest_dir = str(self.config_value("digest_dir")).strip("/")
daily_dir = str(self.config_value("daily_dir")).strip("/")
category_paths = {
bucket: f"{digest_dir}/{bucket.value}" if digest_dir else bucket.value for bucket in _CATEGORY_BUCKETS
}
daily_prefix = f"{daily_dir}/" if daily_dir else ""
digest_paths_by_category = {
label: sorted(path for path in node_by_path if path.startswith(f"{category_path}/"))
for label, category_path in category_paths.items()
}
digest_paths = {path for paths in digest_paths_by_category.values() for path in paths}
edge_keys: set[tuple[str, str, str | None]] = set()
for source in digest_paths:
for link in node_by_path[source].links:
target = link.target_path
if target in digest_paths or (target in node_by_path and target.startswith(daily_prefix)):
edge_keys.add((source, target, link.target_anchor))
daily_paths = {target for _source, target, _anchor in edge_keys if target.startswith(daily_prefix)}
nodes = [
GraphSnapshotNode(
id=f"virtual:{bucket.value}",
path=category_paths[bucket],
name=bucket.value,
indexed=False,
virtual=True,
)
for bucket in _CATEGORY_BUCKETS
]
for bucket in _CATEGORY_BUCKETS:
for path in digest_paths_by_category[bucket]:
edge_keys.add((f"virtual:{bucket.value}", path, None))
for path in sorted(digest_paths | daily_paths):
node = node_by_path[path]
nodes.append(
GraphSnapshotNode(
id=path,
path=path,
name=node.front_matter.name,
description=node.front_matter.description,
indexed=True,
),
)
edges = [
GraphSnapshotEdge(source=source, target=target, target_anchor=anchor)
for source, target, anchor in sorted(edge_keys, key=lambda edge: (edge[0], edge[1], edge[2] or ""))
]
graph = GraphSnapshot(nodes=nodes, edges=edges)
self.context.response.success = True
self.context.response.answer = graph.model_dump()
self.logger.info(f"[{self.name}] nodes={len(nodes)} edges={len(edges)}")
return self.context.response

112
reme/steps/index/search_v2.py
View file

@ -2,7 +2,8 @@
This is the local fork of the upstream search step. It uses
:class:`_ToolContextDedupMixin` for subset-aware interval-merging dedup and
:func:`format_chunks_answer` for session-aware chunk formatting with
:func:`render_chunk_entries` / :func:`join_chunk_entries` for session-aware
chunk formatting with
:data:`ALL_RETURNED_MESSAGE` / :data:`NO_RESULTS_MESSAGE` notices.
"""
@ -12,7 +13,8 @@ import os
from typing import Final
from ._dedup import _ToolContextDedupMixin
from ._source_format import ALL_RETURNED_MESSAGE, NO_RESULTS_MESSAGE, format_chunks_answer
from ._source_format import ALL_RETURNED_MESSAGE, NO_RESULTS_MESSAGE, is_session_path, join_chunk_entries
from ._source_format import merge_session_chunk_intervals, render_chunk_entries
from ..base_step import BaseStep
from ..file_io import extract_daily_date
from ...components import R
@ -116,9 +118,12 @@ class SearchV2Step(_ToolContextDedupMixin, BaseStep):
expand_links_enabled: bool = bool(self.kwargs.get("expand_links", True))
max_links_per_direction: int = int(self.kwargs.get("max_links_per_direction", 10))
tool_context_id: str = (self.context.get("tool_context_id", "") or "").strip()
strict_date_filter: bool = bool(
self.context.get("strict_date_filter") or self.kwargs.get("strict_date_filter", False),
)
# Injected value takes precedence over YAML kwargs; check existence
# (not truthiness) so an explicit False can disable a YAML-true flag.
_strict_date_filter = self.context.get("strict_date_filter")
if _strict_date_filter is None:
_strict_date_filter = self.kwargs.get("strict_date_filter", False)
strict_date_filter: bool = bool(_strict_date_filter)
if not query:
self.context.response.success = False
@ -209,12 +214,15 @@ class SearchV2Step(_ToolContextDedupMixin, BaseStep):
)
dialog_dir = self.config_value("dialog_dir")
self.context.response.answer = format_chunks_answer(
fused,
entries = render_chunk_entries(
merge_session_chunk_intervals(fused, dialog_dir),
dialog_dir,
score_fn=lambda c: self._format_scores(c.scores, hybrid),
link_expansion=link_expansion,
)
if self._session_compress_enabled():
await self._compress_session_entries(entries, query, dialog_dir)
self.context.response.answer = join_chunk_entries(entries)
if not fused:
self.context.response.answer = ALL_RETURNED_MESSAGE if pre_dedup_count > 0 else NO_RESULTS_MESSAGE
self.context.response.metadata["results"] = [
@ -230,3 +238,93 @@ class SearchV2Step(_ToolContextDedupMixin, BaseStep):
if dedup is not None:
self.context.response.metadata["dedup"] = dedup
return self.context.response
def _session_compress_enabled(self) -> bool:
"""True when the injected ``_search._compress.session`` flag is truthy."""
assert self.context is not None
search_cfg: dict = self.context.get("_search") or {}
value = (search_cfg.get("_compress") or {}).get("session")
return value is True or str(value).strip().lower() == "true"
async def _compress_session_entries(self, entries: list[dict[str, str]], query: str, dialog_dir: str) -> None:
"""Compress session-transcript entry bodies in place via the ``compressor`` job.
Only entries whose ``path`` points at a raw session transcript are
compressed; other entries and all non-``body`` fields stay untouched.
When ``_search.type`` is ``query-independent`` the compressor runs
without queries (generic compression); otherwise (``query-aware``,
the default) it receives the injected ``_search.queries`` plus the
current search query.
The compressor receives the already-rendered body (one message per
line) stripped. Its output is adopted whenever the compressor
succeeded and the result is not longer than the input; adopted bodies
get a leading ``compressed session chunk:`` marker so downstream
consumers can tell them from verbatim transcripts.
Degrades gracefully when the ``compressor`` job is missing from the
active config: the whole method becomes a no-op and a warning is
logged, so search behaves as if compression were disabled. This
avoids a hard ``Job compressor not found`` failure when a benchmark
config forgets to define the compressor job/component.
Per-entry exceptions raised by the compressor job (e.g. a temporary
LLM outage) are caught inside ``compress`` so they never propagate
through ``asyncio.gather``: the failing entry keeps its original body
while the remaining entries are still compressed, preserving already
retrieved search results.
"""
assert self.context is not None
# Guard: when the compressor job is missing from the active config
# (e.g. a benchmark config that forgot to define it), degrade
# gracefully to the no-compression behavior instead of raising
# "Job compressor not found" from run_job below.
# Skipped when there is no app_context (e.g. unit tests that mock
# run_job directly), so the mock can still drive compression.
if self.app_context is not None and self.get_job("compressor") is None:
self.logger.warning(
f"[{self.name}] compressor job not found in config; "
"skipping session chunk compression (degrading to uncompressed behavior)",
)
return
search_cfg: dict = self.context.get("_search") or {}
query_type = str(search_cfg.get("type") or "query-aware").strip().lower()
if query_type == "query-independent":
queries: list[str] = []
else:
queries = [str(q).strip() for q in (search_cfg.get("queries") or []) if str(q).strip()]
if query and query not in queries:
queries.append(query)
async def compress(entry: dict[str, str]) -> None:
path = entry.get("path", "")
body = (entry.get("body", "") or "").strip()
if not body:
return
try:
response = await self.run_job("compressor", text=body, queries=queries)
except Exception as exc: # pylint: disable=broad-except
self.logger.warning(
f"[{self.name}] session body compression raised path={path!r} " f"error={exc!r}; keeping original",
)
return
compressed = str(response.answer or "").strip()
if not response.success or not compressed:
self.logger.warning(
f"[{self.name}] session body compression failed path={path!r} "
f"success={response.success} answer={compressed[:100]!r}; keeping original",
)
return
if len(compressed) > len(body):
self.logger.info(
f"[{self.name}] compressed body longer than original "
f"({len(compressed)} > {len(body)}) path={path!r}; keeping original",
)
return
entry["body"] = f"compressed session chunk:\n{compressed}"
targets = [e for e in entries if is_session_path(e.get("path", ""), dialog_dir)]
if not targets:
return
self.logger.info(f"[{self.name}] compressing {len(targets)} session entries with {len(queries)} queries")
await asyncio.gather(*(compress(entry) for entry in targets))

242
reme/steps/index/traverse.py
View file

@ -1,127 +1,181 @@
"""BFS over wikilink edges from one or more seed files.
One record per traversed *edge* (not per node): the same target can repeat
if reached via different anchors or paths. Each record carries the
predecessor and anchor so callers can reconstruct the path. Adjacency is
built once via a single ``file_store.get_nodes()`` call — BFS then runs
purely in memory with no per-frontier round-trips.
"""
"""Build a bounded, frontend-ready graph from indexed Markdown wikilinks."""
from collections import deque
from dataclasses import dataclass
from pathlib import Path
from ..base_step import BaseStep
from ...components import R
from ...schema import FileLink
from ...schema import FileNode, TraverseGraph, TraverseGraphEdge, TraverseGraphNode
_OUT = {"out", "forward", "both"}
_IN = {"in", "backward", "both"}
_VALID = _OUT | _IN
# source path -> list of (neighbor path, link)
Adjacency = dict[str, list[tuple[str, FileLink]]]
_DIRECTION_ALIASES = {
"out": "forward",
"forward": "forward",
"in": "backward",
"backward": "backward",
"both": "both",
}
async def _build_adjacency(file_store) -> tuple[Adjacency, Adjacency]:
"""Single ``get_nodes()`` pass → (outbound, inbound) adjacency maps.
@dataclass(frozen=True, slots=True)
class _TraversalLink:
"""An adjacency entry carrying both traversal and original edge direction."""
Inbound stores the source path next to each link so BFS can attribute
inbound edges back to their origin — ``get_inlinks`` alone returns
target-shaped FileLinks without source attribution.
"""
outbound: Adjacency = {}
inbound: Adjacency = {}
for node in await file_store.get_nodes():
neighbor: str
source: str
target: str
target_anchor: str | None
Adjacency = dict[str, list[_TraversalLink]]
EdgeKey = tuple[str, str, str | None]
def _build_adjacency(nodes: list[FileNode]) -> tuple[Adjacency, Adjacency]:
"""Build forward/backward adjacency while preserving actual wikilink direction."""
forward: Adjacency = {}
backward: Adjacency = {}
for node in nodes:
for link in node.links:
if link.target_path:
outbound.setdefault(node.path, []).append((link.target_path, link))
inbound.setdefault(link.target_path, []).append((node.path, link))
return outbound, inbound
if not link.target_path:
continue
item = _TraversalLink(
neighbor=link.target_path,
source=node.path,
target=link.target_path,
target_anchor=link.target_anchor,
)
forward.setdefault(node.path, []).append(item)
backward.setdefault(link.target_path, []).append(
_TraversalLink(
neighbor=node.path,
source=item.source,
target=item.target,
target_anchor=item.target_anchor,
),
)
return forward, backward
def _bfs(
def _traverse(
seeds: list[str],
max_depth: int,
direction: str,
outbound: Adjacency,
inbound: Adjacency,
) -> list[dict]:
"""In-memory BFS; emits one record per unique (src, dst, anchor) edge."""
sources: list[Adjacency] = []
if direction in _OUT:
sources.append(outbound)
if direction in _IN:
sources.append(inbound)
forward: Adjacency,
backward: Adjacency,
) -> tuple[dict[str, int], dict[EdgeKey, int]]:
"""Return shortest node depths and directed edges reached by bounded BFS."""
adjacency_maps = []
if direction in {"forward", "both"}:
adjacency_maps.append(forward)
if direction in {"backward", "both"}:
adjacency_maps.append(backward)
visited: set[tuple[str, str, str | None]] = set()
results: list[dict] = []
queue: deque[tuple[str, int]] = deque((s, 0) for s in seeds)
node_depths = {seed: 0 for seed in seeds}
edge_depths: dict[EdgeKey, int] = {}
queue = deque(seeds)
while queue:
current, depth = queue.popleft()
if depth >= max_depth:
current = queue.popleft()
current_depth = node_depths[current]
if current_depth >= max_depth:
continue
for src in sources:
for next_path, link in src.get(current, ()):
key = (current, next_path, link.target_anchor)
if key in visited:
continue
visited.add(key)
results.append(
{
"path": next_path,
"depth": depth + 1,
"via": current,
"anchor": link.target_anchor,
},
)
if depth + 1 < max_depth:
queue.append((next_path, depth + 1))
return results
next_depth = current_depth + 1
for adjacency in adjacency_maps:
for link in adjacency.get(current, ()):
edge_key = (link.source, link.target, link.target_anchor)
previous_edge_depth = edge_depths.get(edge_key)
if previous_edge_depth is None or next_depth < previous_edge_depth:
edge_depths[edge_key] = next_depth
previous_node_depth = node_depths.get(link.neighbor)
if previous_node_depth is None or next_depth < previous_node_depth:
node_depths[link.neighbor] = next_depth
queue.append(link.neighbor)
return node_depths, edge_depths
def _build_graph(
*,
seeds: list[str],
max_depth: int,
direction: str,
indexed_nodes: list[FileNode],
node_depths: dict[str, int],
edge_depths: dict[EdgeKey, int],
) -> TraverseGraph:
"""Materialize deterministic public graph schemas from traversal state."""
node_by_path = {node.path: node for node in indexed_nodes}
graph_nodes = []
for path, node_depth in sorted(node_depths.items(), key=lambda item: (item[1], item[0])):
node = node_by_path.get(path)
graph_nodes.append(
TraverseGraphNode(
id=path,
path=path,
name=node.front_matter.name if node is not None else "",
description=node.front_matter.description if node is not None else "",
depth=node_depth,
indexed=node is not None,
),
)
graph_edges = [
TraverseGraphEdge(source=source, target=target, target_anchor=anchor, depth=edge_depth)
for (source, target, anchor), edge_depth in sorted(
edge_depths.items(),
key=lambda item: (item[1], item[0][0], item[0][1], item[0][2] or ""),
)
]
return TraverseGraph(
seeds=seeds,
depth=max_depth,
direction=direction,
nodes=graph_nodes,
edges=graph_edges,
)
@R.register("traverse_step")
class TraverseStep(BaseStep):
"""BFS from one or more seed files to explore wikilink relationships.
Parameters:
path — single seed (str) or list of seeds (workspace-relative).
direction — ``forward`` / ``backward`` / ``both`` (or ``out`` / ``in`` / ``both``).
depth — hop limit (default 1 = immediate neighbors).
"""
"""Return a bounded wikilink graph rooted at one or more workspace paths."""
async def execute(self):
assert self.context is not None
raw = self.context.get("path")
items = [raw] if isinstance(raw, (str, Path)) else list(raw or [])
seeds = [str(p) for p in items if p]
assert seeds, "path is required"
depth = int(self.context.get("depth") or 1)
direction = (self.context.get("direction") or "both").lower()
assert direction in _VALID, f"direction must be one of {sorted(_VALID)}, got {direction!r}"
raw_paths = self.context.get("path")
items = [raw_paths] if isinstance(raw_paths, (str, Path)) else list(raw_paths or [])
seeds = list(dict.fromkeys(str(path).replace("\\", "/") for path in items if path))
if not seeds:
raise ValueError("path is required")
outbound, inbound = await _build_adjacency(self.file_store)
results = _bfs(seeds, depth, direction, outbound, inbound)
raw_depth = self.context.get("depth")
max_depth = 1 if raw_depth is None else int(raw_depth)
if max_depth < 0:
raise ValueError("depth must be greater than or equal to 0")
self.logger.info(
f"[{self.name}] seeds={seeds!r} depth={depth} direction={direction} "
f"nodes={len(outbound) + len(inbound)} edges={len(results)}",
raw_direction = str(self.context.get("direction") or "both").lower()
direction = _DIRECTION_ALIASES.get(raw_direction)
if direction is None:
raise ValueError(f"direction must be one of {sorted(_DIRECTION_ALIASES)}, got {raw_direction!r}")
indexed_nodes = await self.file_store.get_nodes()
forward, backward = _build_adjacency(indexed_nodes)
node_depths, edge_depths = _traverse(seeds, max_depth, direction, forward, backward)
graph = _build_graph(
seeds=seeds,
max_depth=max_depth,
direction=direction,
indexed_nodes=indexed_nodes,
node_depths=node_depths,
edge_depths=edge_depths,
)
label = seeds[0] if len(seeds) == 1 else f"{len(seeds)} seeds"
if not results:
answer = f"No edges found from {label}"
else:
header = f"Traversed {len(results)} edge(s) from {label}"
lines = [header, ""]
for r in results:
target = r["path"]
if r["anchor"]:
target = f"{target}#{r['anchor']}"
lines.append(f"[depth={r['depth']}] {r['via']} --> {target}")
answer = "\n".join(lines)
self.context.response.success = True
self.context.response.answer = answer
self.context.response.metadata.update({"edges": results, "count": len(results)})
self.context.response.answer = graph.model_dump()
self.logger.info(
f"[{self.name}] seeds={seeds!r} depth={max_depth} direction={direction} "
f"nodes={len(graph.nodes)} edges={len(graph.edges)}",
)
return self.context.response

8
reme/steps/index/vector_search.py
View file

@ -3,7 +3,8 @@
from typing import Final
from ._dedup import _ToolContextDedupMixin
from ._source_format import ALL_RETURNED_MESSAGE, NO_RESULTS_MESSAGE, format_chunks_answer
from ._source_format import ALL_RETURNED_MESSAGE, NO_RESULTS_MESSAGE, join_chunk_entries
from ._source_format import merge_session_chunk_intervals, render_chunk_entries
from ..base_step import BaseStep
from ...components import R
@ -48,11 +49,12 @@ class VectorSearchStep(_ToolContextDedupMixin, BaseStep):
results = results[:limit]
dialog_dir = self.config_value("dialog_dir")
self.context.response.answer = format_chunks_answer(
results,
entries = render_chunk_entries(
merge_session_chunk_intervals(results, dialog_dir),
dialog_dir,
include_source=self.include_source,
)
self.context.response.answer = join_chunk_entries(entries)
if not results:
self.context.response.answer = ALL_RETURNED_MESSAGE if pre_dedup_count > 0 else NO_RESULTS_MESSAGE
self.context.response.metadata["results"] = [

281
scripts/github_star_growth.py Executable file
View file

@ -0,0 +1,281 @@
#!/usr/bin/env python3
"""Report daily new GitHub stars for a repository using the GitHub CLI."""
from __future__ import annotations
import argparse
import csv
import json
import math
import subprocess
import sys
from collections import Counter
from datetime import UTC, date, datetime, timedelta
from pathlib import Path
from typing import Any, TextIO
from xml.sax.saxutils import escape
DEFAULT_REPOSITORY = "agentscope-ai/ReMe"
QUERY = """
query($owner: String!, $name: String!, $before: String) {
repository(owner: $owner, name: $name) {
nameWithOwner
stargazerCount
stargazers(last: 100, before: $before) {
edges {
starredAt
}
pageInfo {
hasPreviousPage
startCursor
}
}
}
}
"""
def parse_args() -> argparse.Namespace:
"""Parse and validate command-line arguments."""
parser = argparse.ArgumentParser(
description="Print daily new-star counts for the latest N UTC calendar days.",
)
parser.add_argument(
"--repo",
default=DEFAULT_REPOSITORY,
metavar="OWNER/REPO",
help=f"GitHub repository (default: {DEFAULT_REPOSITORY})",
)
parser.add_argument(
"--days",
type=int,
default=365,
help="number of UTC calendar days to include (default: 365)",
)
parser.add_argument(
"--output",
type=Path,
metavar="PATH",
help="write CSV to PATH instead of standard output",
)
parser.add_argument(
"--chart",
type=Path,
metavar="PATH",
help="write the SVG chart to PATH (default: CSV path with an .svg suffix)",
)
args = parser.parse_args()
if args.days < 1:
parser.error("--days must be at least 1")
if args.repo.count("/") != 1 or any(not part for part in args.repo.split("/")):
parser.error("--repo must have the form OWNER/REPO")
return args
def query_github(owner: str, name: str, before: str | None) -> dict[str, Any]:
"""Fetch one page of repository stargazers through ``gh api graphql``."""
request = {
"query": QUERY,
"variables": {"owner": owner, "name": name, "before": before},
}
try:
result = subprocess.run(
["gh", "api", "graphql", "--input", "-"],
input=json.dumps(request),
text=True,
capture_output=True,
check=False,
)
except FileNotFoundError as exc:
raise RuntimeError("GitHub CLI 'gh' is not installed or is not on PATH") from exc
if result.returncode != 0:
detail = result.stderr.strip() or result.stdout.strip()
raise RuntimeError(f"gh api graphql failed: {detail}")
try:
payload = json.loads(result.stdout)
except json.JSONDecodeError as exc:
raise RuntimeError("gh returned invalid JSON") from exc
if payload.get("errors"):
messages = "; ".join(error.get("message", str(error)) for error in payload["errors"])
raise RuntimeError(f"GitHub GraphQL API error: {messages}")
repository = payload.get("data", {}).get("repository")
if repository is None:
raise RuntimeError(f"repository not found or inaccessible: {owner}/{name}")
return repository
def fetch_daily_stars(repository: str, start_date: date) -> tuple[Counter[date], int]:
"""Fetch current stargazers and group their star timestamps by UTC date."""
owner, name = repository.split("/", maxsplit=1)
daily_stars: Counter[date] = Counter()
before: str | None = None
total_stars = 0
while True:
data = query_github(owner, name, before)
total_stars = data["stargazerCount"]
connection = data["stargazers"]
edges = connection["edges"]
reached_start = False
for edge in edges:
starred_at = datetime.fromisoformat(edge["starredAt"].replace("Z", "+00:00"))
starred_date = starred_at.astimezone(UTC).date()
if starred_date < start_date:
reached_start = True
continue
daily_stars[starred_date] += 1
page_info = connection["pageInfo"]
if reached_start or not page_info["hasPreviousPage"]:
break
before = page_info["startCursor"]
if before is None:
raise RuntimeError("GitHub returned an invalid pagination cursor")
return daily_stars, total_stars
def write_csv(stream: TextIO, start_date: date, days: int, counts: Counter[date]) -> None:
"""Write a row for every date in the requested period, including zero days."""
writer = csv.writer(stream, lineterminator="\n")
writer.writerow(["date", "new_stars"])
for offset in range(days):
current_date = start_date + timedelta(days=offset)
writer.writerow([current_date.isoformat(), counts[current_date]])
def nice_tick_step(maximum: int, tick_count: int = 5) -> int:
"""Return a readable positive interval for numeric axis ticks."""
rough_step = max(maximum, 1) / tick_count
magnitude = 10 ** math.floor(math.log10(rough_step))
normalized = rough_step / magnitude
if normalized <= 1:
multiplier = 1
elif normalized <= 2:
multiplier = 2
elif normalized <= 5:
multiplier = 5
else:
multiplier = 10
return max(1, multiplier * magnitude)
def write_svg_chart(path: Path, repository: str, dates: list[date], values: list[int]) -> None:
"""Render daily star increments as a dependency-free SVG bar chart."""
width, height = 1280, 640
margin_left, margin_right = 75, 30
margin_top, margin_bottom = 70, 85
plot_width = width - margin_left - margin_right
plot_height = height - margin_top - margin_bottom
tick_step = nice_tick_step(max(values, default=0))
axis_max = max(tick_step, math.ceil(max(values, default=0) / tick_step) * tick_step)
bar_slot = plot_width / len(values)
bar_width = max(0.8, bar_slot * 0.82)
svg = [
'<?xml version="1.0" encoding="UTF-8"?>',
(
f'<svg xmlns="http://www.w3.org/2000/svg" width="{width}" height="{height}" '
f'viewBox="0 0 {width} {height}" role="img">'
),
f"<title>{escape(repository)} daily new GitHub stars</title>",
'<rect width="100%" height="100%" fill="#ffffff"/>',
(
f'<text x="{width / 2}" y="34" text-anchor="middle" font-family="sans-serif" '
f'font-size="22" font-weight="600" fill="#24292f">{escape(repository)} daily new stars</text>'
),
]
for tick in range(0, axis_max + 1, tick_step):
y = margin_top + plot_height - (tick / axis_max * plot_height)
svg.append(
f'<line x1="{margin_left}" y1="{y:.2f}" x2="{width - margin_right}" y2="{y:.2f}" '
'stroke="#d8dee4" stroke-width="1"/>',
)
svg.append(
f'<text x="{margin_left - 10}" y="{y + 4:.2f}" text-anchor="end" font-family="sans-serif" '
f'font-size="12" fill="#57606a">{tick}</text>',
)
for index, (current_date, value) in enumerate(zip(dates, values, strict=True)):
bar_height = value / axis_max * plot_height
x = margin_left + index * bar_slot + (bar_slot - bar_width) / 2
y = margin_top + plot_height - bar_height
svg.append(
f'<rect x="{x:.2f}" y="{y:.2f}" width="{bar_width:.2f}" height="{bar_height:.2f}" '
f'fill="#2f81f7"><title>{current_date.isoformat()}: {value} new stars</title></rect>',
)
label_count = min(12, len(dates))
label_indexes = sorted({round(index * (len(dates) - 1) / max(label_count - 1, 1)) for index in range(label_count)})
baseline = margin_top + plot_height
for index in label_indexes:
x = margin_left + (index + 0.5) * bar_slot
svg.append(
f'<line x1="{x:.2f}" y1="{baseline}" x2="{x:.2f}" y2="{baseline + 5}" stroke="#57606a"/>',
)
svg.append(
f'<text x="{x:.2f}" y="{baseline + 20}" text-anchor="end" '
f'transform="rotate(-35 {x:.2f} {baseline + 20})" font-family="sans-serif" '
f'font-size="11" fill="#57606a">{dates[index].isoformat()}</text>',
)
svg.extend(
[
(
f'<line x1="{margin_left}" y1="{baseline}" x2="{width - margin_right}" y2="{baseline}" '
'stroke="#57606a"/>'
),
(
f'<text x="18" y="{margin_top + plot_height / 2}" text-anchor="middle" '
f'transform="rotate(-90 18 {margin_top + plot_height / 2})" font-family="sans-serif" '
'font-size="13" fill="#24292f">New stars</text>'
),
"</svg>",
],
)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text("\n".join(svg) + "\n", encoding="utf-8")
def main() -> int:
"""Run the report and return a process exit code."""
args = parse_args()
end_date = datetime.now(UTC).date()
start_date = end_date - timedelta(days=args.days - 1)
try:
counts, total_stars = fetch_daily_stars(args.repo, start_date)
dates = [start_date + timedelta(days=offset) for offset in range(args.days)]
values = [counts[current_date] for current_date in dates]
if args.output:
args.output.parent.mkdir(parents=True, exist_ok=True)
with args.output.open("w", encoding="utf-8", newline="") as stream:
write_csv(stream, start_date, args.days, counts)
else:
write_csv(sys.stdout, start_date, args.days, counts)
chart_path = args.chart or (args.output.with_suffix(".svg") if args.output else Path("star_growth.svg"))
write_svg_chart(chart_path, args.repo, dates, values)
except (OSError, RuntimeError) as exc:
print(f"error: {exc}", file=sys.stderr)
return 1
destination = str(args.output) if args.output else "stdout"
period_stars = sum(counts.values())
print(
f"Fetched {period_stars} current stargazers in {start_date}..{end_date}; "
f"repository currently has {total_stars} stars. CSV: {destination}; chart: {chart_path}",
file=sys.stderr,
)
return 0
if __name__ == "__main__":
raise SystemExit(main())

44
tests/unit/test_app_config_step.py Normal file
View file

@ -0,0 +1,44 @@
"""Tests for the application config step."""
import asyncio
from reme.components.application_context import ApplicationContext
from reme.steps.common.app_config import AppConfigStep
def test_app_config_step_returns_effective_config_without_secrets(tmp_path):
"""Effective values are returned while startup secrets stay private."""
context = ApplicationContext(
app_name="Test ReMe",
workspace_dir=str(tmp_path),
environment={"PRIVATE_VALUE": "hidden"},
components={
"as_llm": {
"default": {
"backend": "openai",
"credential": {"api_key": "key", "base_url": "https://example.test"},
},
},
},
mcp_servers={
"private": {
"url": "https://user:password@example.test/mcp?access_token=url-token&mode=read",
"headers": {
"Authorization": "Bearer header-token",
"X-API-Key": "header-key",
},
"client_secret": "client-secret",
},
},
)
response = asyncio.run(AppConfigStep(app_context=context)())
assert response.answer["app_name"] == "Test ReMe"
assert "environment" not in response.answer
credential = response.answer["components"]["as_llm"]["default"]["credential"]
assert credential == {"api_key": "***", "base_url": "https://example.test"}
mcp_server = response.answer["mcp_servers"]["private"]
assert mcp_server["url"] == "https://user:***@example.test/mcp?access_token=%2A%2A%2A&mode=read"
assert mcp_server["headers"] == {"Authorization": "***", "X-API-Key": "***"}
assert mcp_server["client_secret"] == "***"

128
tests/unit/test_beam_auto_memory.py Normal file
View file

@ -0,0 +1,128 @@
"""Unit tests for BEAM auto-memory segmented ingestion and line numbering."""
# pylint: disable=missing-class-docstring,missing-function-docstring,protected-access
from agentscope.message import Msg
from reme.components.runtime_context import RuntimeContext
from reme.steps.benchmark.beam.auto_memory import (
BeamAutoMemoryStep,
split_turn_segments,
)
def _msg(role: str, words: int, created_at: str = "2024-03-01T09:00:00") -> Msg:
return Msg(
name=role,
role=role,
content=[{"type": "text", "text": " ".join(["w"] * words)}],
created_at=created_at,
)
def _dialog(n_turns: int, words_per_msg: int) -> list[Msg]:
messages: list[Msg] = []
for _ in range(n_turns):
messages.append(_msg("user", words_per_msg))
messages.append(_msg("assistant", words_per_msg))
return messages
class TestSplitTurnSegments:
def test_empty(self):
assert not split_turn_segments([], 100)
def test_disabled_returns_single_segment(self):
messages = _dialog(3, 10)
segments = split_turn_segments(messages, 0)
assert len(segments) == 1
assert segments[0] == (0, messages)
def test_under_limit_single_segment(self):
messages = _dialog(3, 10) # 60 words total
segments = split_turn_segments(messages, 100)
assert len(segments) == 1
assert segments[0][0] == 0
assert segments[0][1] == messages
def test_splits_at_turn_boundaries(self):
# 4 turns x 20 words each; limit 40 -> 2 turns per segment
messages = _dialog(4, 10)
segments = split_turn_segments(messages, 40)
assert len(segments) == 2
offsets = [offset for offset, _ in segments]
assert offsets == [0, 4]
# Every segment starts with a user message and ends with an assistant
for _, segment in segments:
assert segment[0].role == "user"
assert segment[-1].role == "assistant"
# No message lost or duplicated, order preserved
flattened = [m for _, segment in segments for m in segment]
assert flattened == messages
def test_never_splits_inside_a_turn(self):
# One turn alone exceeds the limit -> becomes its own oversized segment
messages = [
_msg("user", 5),
_msg("assistant", 5),
_msg("user", 50),
_msg("assistant", 50), # 100-word turn > limit 60
_msg("user", 5),
_msg("assistant", 5),
]
segments = split_turn_segments(messages, 60)
assert [offset for offset, _ in segments] == [0, 2, 4]
assert [len(segment) for _, segment in segments] == [2, 2, 2]
def test_offsets_are_original_indices(self):
messages = _dialog(5, 30) # 60 words per turn
segments = split_turn_segments(messages, 120)
# 2 turns per segment -> offsets 0, 4, 8
assert [offset for offset, _ in segments] == [0, 4, 8]
for offset, segment in segments:
for i, msg in enumerate(segment):
assert msg is messages[offset + i]
def test_multi_assistant_turn_stays_together(self):
messages = [
_msg("user", 10),
_msg("assistant", 10),
_msg("assistant", 10),
_msg("user", 10),
_msg("assistant", 10),
]
segments = split_turn_segments(messages, 30)
assert [offset for offset, _ in segments] == [0, 3]
assert len(segments[0][1]) == 3
class TestFormatHistoryLineNumbers:
def _step(self, offset: int) -> BeamAutoMemoryStep:
step = BeamAutoMemoryStep(name="beam_auto_memory_step", backend="beam_auto_memory_step")
step.context = RuntimeContext(
session_id="beam_1M_1_batch1",
beam_line_offset=offset,
)
return step
def test_numbers_start_at_one_without_offset(self):
step = self._step(0)
history = step._format_history(_dialog(2, 3))
assert "[L1 | user @" in history
assert "[L4 | assistant @" in history
assert "lines 1-4" in history
assert "session/dialog/beam_1M_1_batch1.jsonl" in history
def test_numbers_use_original_file_offset(self):
step = self._step(40)
history = step._format_history(_dialog(2, 3))
assert "[L41 | user @" in history
assert "[L44 | assistant @" in history
assert "lines 41-44" in history
assert "[L1 " not in history
def test_build_messages_passes_msg_objects_through(self):
step = self._step(0)
messages = _dialog(2, 3)
rebuilt = step._build_messages(messages)
assert [m.id for m in rebuilt] == [m.id for m in messages]

136
tests/unit/test_common_steps.py
View file

@ -10,7 +10,7 @@ import warnings
from reme.components.agent_wrapper import BaseAgentWrapper
from reme.components.application_context import ApplicationContext
from reme.components.file_store import LocalFileStore
from reme.schema import FileLink, FileNode
from reme.schema import FileFrontMatter, FileLink, FileNode, TraverseGraph
from reme.steps.common.add import AddStep
from reme.steps.common.health_check import _file_graph_status
from reme.steps.common.llm_demo import LLMDemoStep
@ -42,7 +42,13 @@ def _run(coro):
asyncio.run(coro)
def _node(path: str, links: list[tuple[str, str | None]] | None = None) -> FileNode:
def _node(
path: str,
links: list[tuple[str, str | None]] | None = None,
*,
name: str = "",
description: str = "",
) -> FileNode:
"""Build a FileNode with (target_path, target_anchor) outgoing edges."""
return FileNode(
path=path,
@ -50,6 +56,7 @@ def _node(path: str, links: list[tuple[str, str | None]] | None = None) -> FileN
links=[
FileLink(source_path=path, target_path=target, target_anchor=anchor) for target, anchor in (links or [])
],
front_matter=FileFrontMatter(name=name, description=description),
)
@ -63,7 +70,7 @@ async def _make_store(nodes: list[FileNode]) -> LocalFileStore:
def _edges(step) -> list[dict]:
return step.context.response.metadata.get("edges", [])
return step.context.response.answer["edges"]
def test_add_step_coerces_numeric_inputs():
@ -207,12 +214,11 @@ def test_traverse_forward_depth_1():
step = traverse_mod.TraverseStep(file_store=store)
await step(path="a.md", direction="forward", depth=1)
results = _edges(step)
paths = {r["path"] for r in results}
paths = {r["target"] for r in results}
assert paths == {"b.md", "c.md"}
# Anchors remain part of traversal metadata.
c_edge = next(r for r in results if r["path"] == "c.md")
assert c_edge["anchor"] == "intro"
assert c_edge["via"] == "a.md"
c_edge = next(r for r in results if r["target"] == "c.md")
assert c_edge["target_anchor"] == "intro"
assert c_edge["source"] == "a.md"
assert c_edge["depth"] == 1
await store.close()
print("✓ test_traverse_forward_depth_1 passed")
@ -220,6 +226,25 @@ def test_traverse_forward_depth_1():
asyncio.run(run())
def test_traverse_normalizes_windows_seed_path():
"""Windows-style seeds match the graph's portable POSIX path keys."""
async def run():
with tempfile.TemporaryDirectory() as tmp, _temp_chdir(tmp):
store = await _make_store(
[
_node("topics/a.md", [("topics/b.md", None)]),
_node("topics/b.md"),
],
)
step = traverse_mod.TraverseStep(file_store=store)
await step(path=r"topics\a.md", direction="forward", depth=1)
assert [edge["target"] for edge in _edges(step)] == ["topics/b.md"]
await store.close()
asyncio.run(run())
def test_traverse_backward_returns_inlinks():
"""direction=backward walks inbound edges."""
@ -235,7 +260,10 @@ def test_traverse_backward_returns_inlinks():
step = traverse_mod.TraverseStep(file_store=store)
await step(path="b.md", direction="backward", depth=1)
results = _edges(step)
assert {r["path"] for r in results} == {"a.md", "c.md"}
assert {(r["source"], r["target"]) for r in results} == {
("a.md", "b.md"),
("c.md", "b.md"),
}
await store.close()
print("✓ test_traverse_backward_returns_inlinks passed")
@ -257,7 +285,7 @@ def test_traverse_depth_2_expands():
step = traverse_mod.TraverseStep(file_store=store)
await step(path="a.md", direction="forward", depth=2)
results = _edges(step)
depth_map = {r["path"]: r["depth"] for r in results}
depth_map = {r["target"]: r["depth"] for r in results}
assert depth_map.get("b.md") == 1
assert depth_map.get("c.md") == 2
await store.close()
@ -320,13 +348,96 @@ def test_traverse_both_directions():
step = traverse_mod.TraverseStep(file_store=store)
await step(path="center.md", direction="both", depth=1)
results = _edges(step)
assert {r["path"] for r in results} == {"upstream.md", "downstream.md"}
assert {(r["source"], r["target"]) for r in results} == {
("upstream.md", "center.md"),
("center.md", "downstream.md"),
}
await store.close()
print("✓ test_traverse_both_directions passed")
asyncio.run(run())
def test_traverse_both_preserves_reciprocal_edge_directions():
"""Opposite wikilinks remain two distinct directed graph edges."""
async def run():
with tempfile.TemporaryDirectory() as tmp, _temp_chdir(tmp):
store = await _make_store(
[
_node("a.md", [("b.md", None)]),
_node("b.md", [("a.md", None)]),
],
)
step = traverse_mod.TraverseStep(file_store=store)
response = await step(path="a.md", direction="both", depth=1)
graph = TraverseGraph.model_validate(response.answer)
assert {(edge.source, edge.target) for edge in graph.edges} == {
("a.md", "b.md"),
("b.md", "a.md"),
}
assert response.metadata == {}
await store.close()
asyncio.run(run())
def test_traverse_returns_frontmatter_and_unresolved_nodes():
"""Graph nodes expose labels and distinguish indexed files from dangling targets."""
async def run():
with tempfile.TemporaryDirectory() as tmp, _temp_chdir(tmp):
store = await _make_store(
[
_node(
"a.md",
[("missing.md", "details")],
name="Alpha",
description="Root note",
),
],
)
step = traverse_mod.TraverseStep(file_store=store)
response = await step(path="a.md", direction="forward", depth=1)
graph = TraverseGraph.model_validate(response.answer)
nodes = {node.path: node for node in graph.nodes}
assert graph.version == 1
assert graph.seeds == ["a.md"]
assert nodes["a.md"].model_dump() == {
"id": "a.md",
"path": "a.md",
"name": "Alpha",
"description": "Root note",
"depth": 0,
"indexed": True,
}
assert nodes["missing.md"].indexed is False
assert nodes["missing.md"].depth == 1
assert graph.edges[0].target_anchor == "details"
await store.close()
asyncio.run(run())
def test_traverse_depth_zero_returns_only_seed_nodes():
"""A zero-hop traversal is valid and emits no edges."""
async def run():
with tempfile.TemporaryDirectory() as tmp, _temp_chdir(tmp):
store = await _make_store([_node("a.md", [("b.md", None)]), _node("b.md")])
step = traverse_mod.TraverseStep(file_store=store)
response = await step(path="a.md", direction="forward", depth=0)
graph = TraverseGraph.model_validate(response.answer)
assert [node.path for node in graph.nodes] == ["a.md"]
assert graph.edges == []
await store.close()
asyncio.run(run())
if __name__ == "__main__":
print("\n=== traverse step tests ===")
test_traverse_forward_depth_1()
@ -335,4 +446,7 @@ if __name__ == "__main__":
test_traverse_short_seed_yields_empty()
test_traverse_not_found_seed()
test_traverse_both_directions()
test_traverse_both_preserves_reciprocal_edge_directions()
test_traverse_returns_frontmatter_and_unresolved_nodes()
test_traverse_depth_zero_returns_only_seed_nodes()
print("\n所有测试通过!")

72
tests/unit/test_compressor_step.py Normal file
View file

@ -0,0 +1,72 @@
"""Unit tests for CompressorStep without LLM dependencies."""
import asyncio
from agentscope.message import TextBlock, ThinkingBlock
from agentscope.model import ChatResponse
from reme.steps.evolve.compressor import CompressorStep
def test_response_text_handles_non_streaming_chat_response():
"""A plain ChatResponse must not crash the async-iteration check.
Regression: ``ChatResponse`` inherits agentscope's ``DictMixin`` whose
instance ``__getattr__`` raises ``KeyError``, so ``hasattr(result,
"__aiter__")`` raised instead of returning False and every compression
failed with ``KeyError: '__aiter__'``.
"""
async def run():
result = ChatResponse(
content=[
{"type": "thinking", "thinking": "internal reasoning"},
{"type": "text", "text": "compressed "},
{"type": "text", "text": "output"},
],
is_last=True,
)
text = await CompressorStep._response_text(result) # pylint: disable=protected-access
assert text == "compressed output"
asyncio.run(run())
def test_response_text_handles_pydantic_content_blocks():
"""Real LLM responses carry pydantic block models, not dicts.
Regression: ``block.get(...)`` crashed with ``'ThinkingBlock' object has
no attribute 'get'`` on responses from reasoning models.
"""
async def run():
result = ChatResponse(
content=[
ThinkingBlock(type="thinking", thinking="internal reasoning"),
TextBlock(type="text", text="compressed output"),
],
is_last=True,
)
text = await CompressorStep._response_text(result) # pylint: disable=protected-access
assert text == "compressed output"
asyncio.run(run())
def test_response_text_consumes_streaming_chunks_and_keeps_last():
"""An async generator of ChatResponses is drained and the last chunk wins."""
async def run():
async def stream():
yield ChatResponse(content=[{"type": "text", "text": "partial"}], is_last=False)
yield ChatResponse(content=[{"type": "text", "text": "final text"}], is_last=True)
text = await CompressorStep._response_text(stream()) # pylint: disable=protected-access
assert text == "final text"
asyncio.run(run())

18
tests/unit/test_config_parser.py
View file

@ -49,6 +49,24 @@ def test_default_config_registers_daily_write_job():
assert job["parameters"]["required"] == ["name", "description", "session_id", "content"]
def test_default_config_registers_app_config_job():
"""The default service exposes the effective application config."""
job = _load_config("default.yaml")["jobs"]["app_config"]
assert job["backend"] == "base"
assert job["steps"] == [{"backend": "app_config_step"}]
def test_default_config_registers_workspace_web_jobs():
"""The local web client has stable save and streaming chat actions."""
jobs = _load_config("default.yaml")["jobs"]
assert jobs["save"]["steps"] == [{"backend": "save_step"}]
assert jobs["load"]["steps"] == [{"backend": "load_step", "max_bytes": 5242880}]
assert jobs["chat"]["backend"] == "stream"
assert jobs["chat"]["steps"] == [{"backend": "chat_step", "agent_wrapper": "default"}]
def test_default_config_keeps_frontmatter_chunk_metadata_opt_in():
"""Markdown frontmatter-to-chunk metadata is disabled by default for compatibility."""
cfg = _load_config("default.yaml")

43
tests/unit/test_crud_steps.py
View file

@ -30,7 +30,7 @@ import asyncio
import os
import tempfile
import warnings
from pathlib import Path
from pathlib import Path, PureWindowsPath
from reme.components.file_store import LocalFileStore
from reme.schema import FileNode
@ -177,6 +177,14 @@ def test_list_lists_files():
asyncio.run(run())
def test_list_formats_workspace_relative_windows_paths_as_posix():
"""Workspace-relative list results use the graph's portable path format."""
workspace = PureWindowsPath("C:/workspace")
files = [workspace / "topics" / "a.md"]
assert crud_list.ListStep._format_relative(files, workspace) == ["topics/a.md"]
def test_list_empty_directory_has_explicit_answer():
"""list tells an LLM explicitly when the target directory has no files."""
@ -770,6 +778,39 @@ def test_read_empty_path_rejected():
_run(run())
def test_read_format_session_injected_false_disables_yaml_true():
"""Runtime-injected ``read_step_format_session=False`` takes precedence
over a YAML-true config — existence check, not truthiness."""
async def run():
from agentscope.message import Msg
m = Msg(name="user", role="user", content=[{"type": "text", "text": "hello world"}])
jsonl_body = m.model_dump_json() + "\n"
with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp):
session_path = Path(tmp) / "session" / "dialog" / "s.jsonl"
session_path.parent.mkdir(parents=True, exist_ok=True)
session_path.write_text(jsonl_body, encoding="utf-8")
store = await _make_store()
# YAML config read_step_format_session=True, no injection → formatting active.
step = crud_read.ReadStep(file_store=store, read_step_format_session=True)
await step(path="session/dialog/s.jsonl")
assert "[user @" in str(step.context.response.answer)
# Same YAML config, but runtime injects False → must disable formatting.
step = crud_read.ReadStep(file_store=store, read_step_format_session=True)
await step(path="session/dialog/s.jsonl", read_step_format_session=False)
assert "[user @" not in str(step.context.response.answer)
await store.close()
print("✓ test_read_format_session_injected_false_disables_yaml_true passed")
_run(run())
# -- write / edit --------------------------------------------------------

94
tests/unit/test_graph_snapshot_step.py Normal file
View file

@ -0,0 +1,94 @@
"""Tests for the frontend-ready category-rooted digest graph snapshot."""
import asyncio
from reme.components.file_store import LocalFileStore
from reme.schema import FileFrontMatter, FileLink, FileNode, GraphSnapshot
from reme.steps.index.graph_snapshot import GraphSnapshotStep
def _node(
path: str,
links: list[tuple[str, str | None]] | None = None,
*,
name: str = "",
description: str = "",
) -> FileNode:
return FileNode(
path=path,
st_mtime=1.0,
links=[
FileLink(source_path=path, target_path=target, target_anchor=anchor) for target, anchor in (links or [])
],
front_matter=FileFrontMatter(name=name, description=description),
)
def test_graph_snapshot_returns_category_roots_digest_links_and_daily_leaves(tmp_path, monkeypatch):
"""The snapshot is rooted at digest categories and stops at indexed daily notes."""
monkeypatch.chdir(tmp_path)
async def run():
store = LocalFileStore(name="snapshot", embedding_store="")
await store.start()
try:
await store.file_graph.upsert_nodes(
[
_node(
"digest/wiki/alpha.md",
[
("digest/personal/beta.md", "intro"),
("daily/2026-08-05/event.md", None),
("daily/2026-08-05/missing.md", None),
("resource/ignored.md", None),
],
name="Alpha",
description="Root note",
),
_node("digest/personal/beta.md", [("digest/wiki/alpha.md", None)], name="Beta"),
_node("digest/procedure/how-to.md", name="How to"),
_node(
"daily/2026-08-05/event.md",
[("digest/procedure/how-to.md", None), ("daily/2026-08-05/hidden.md", None)],
name="Event",
),
_node("daily/2026-08-05/isolated.md", name="Isolated daily"),
_node("resource/ignored.md", name="Ignored resource"),
],
)
response = await GraphSnapshotStep(file_store=store)()
graph = GraphSnapshot.model_validate(response.answer)
assert response.success is True
assert response.metadata == {}
assert [node.id for node in graph.nodes[:3]] == [
"virtual:wiki",
"virtual:personal",
"virtual:procedure",
]
assert [node.path for node in graph.nodes[3:]] == [
"daily/2026-08-05/event.md",
"digest/personal/beta.md",
"digest/procedure/how-to.md",
"digest/wiki/alpha.md",
]
nodes = {node.id: node for node in graph.nodes}
assert nodes["virtual:wiki"].path == "digest/wiki"
assert nodes["virtual:procedure"].path == "digest/procedure"
assert nodes["virtual:wiki"].virtual is True
assert nodes["digest/wiki/alpha.md"].name == "Alpha"
assert nodes["digest/wiki/alpha.md"].description == "Root note"
assert nodes["digest/wiki/alpha.md"].virtual is False
assert {(edge.source, edge.target, edge.target_anchor) for edge in graph.edges} == {
("virtual:wiki", "digest/wiki/alpha.md", None),
("virtual:personal", "digest/personal/beta.md", None),
("virtual:procedure", "digest/procedure/how-to.md", None),
("digest/wiki/alpha.md", "digest/personal/beta.md", "intro"),
("digest/wiki/alpha.md", "daily/2026-08-05/event.md", None),
("digest/personal/beta.md", "digest/wiki/alpha.md", None),
}
finally:
await store.close()
asyncio.run(run())

277
tests/unit/test_search_step.py
View file

@ -2,11 +2,13 @@
import asyncio
from agentscope.message import Msg
from reme.components.file_store import BaseFileStore
from reme.components import ApplicationContext
from reme.components.runtime_context import RuntimeContext
from reme.enumeration import LinkScopeEnum
from reme.schema import FileChunk, FileLink, FileNode
from reme.schema import FileChunk, FileLink, FileNode, Response
from reme.steps.index import (
AddDraftStep,
Bm25SearchStep,
@ -984,3 +986,276 @@ def test_search_step_non_strict_date_filter_not_in_search_filter():
assert "strict_date_filter" not in sf
asyncio.run(run())
def test_search_v2_step_compresses_session_bodies_when_injected_flag_set():
"""With ``_search._compress.session`` truthy, session bodies are replaced by the
compressor job output while headers and non-session entries stay untouched."""
async def run():
session = _chunk("s1", "session/dialog/s1.jsonl", "hello world dialog text", "vector", 0.9, line=2)
note = _chunk("n1", "daily/a.md", "note text", "vector", 0.8)
store = FakeSearchStore(vector_results=[session, note])
step = SearchV2Step(file_store=store, expand_links=False)
calls: list[dict] = []
async def fake_run_job(name, /, **kwargs):
calls.append({"name": name, **kwargs})
return Response(success=True, answer="compressed!")
step.run_job = fake_run_job
ctx = RuntimeContext(
query="alpha",
limit=5,
_search={"_compress": {"session": "true"}, "queries": ["orig question"]},
)
resp = await step(ctx)
assert resp.success is True
assert "compressed!" in resp.answer
assert "hello world dialog text" not in resp.answer
assert "note text" in resp.answer
assert "session/dialog/s1.jsonl:2-2" in resp.answer
assert calls == [
{"name": "compressor", "text": "hello world dialog text", "queries": ["orig question", "alpha"]},
]
asyncio.run(run())
def test_search_v2_step_query_independent_compression_passes_no_queries():
"""With ``_search.type`` = ``query-independent`` the compressor receives no queries;
any other (or missing) type keeps the query-aware behavior."""
async def run():
session = _chunk("s1", "session/dialog/s1.jsonl", "hello world dialog text", "vector", 0.9)
store = FakeSearchStore(vector_results=[session])
step = SearchV2Step(file_store=store, expand_links=False)
calls: list[dict] = []
async def fake_run_job(name, /, **kwargs):
calls.append({"name": name, **kwargs})
return Response(success=True, answer="compressed!")
step.run_job = fake_run_job
ctx = RuntimeContext(
query="alpha",
limit=5,
_search={"_compress": {"session": "true"}, "queries": ["orig question"], "type": "query-independent"},
)
resp = await step(ctx)
assert "compressed!" in resp.answer
assert calls == [{"name": "compressor", "text": "hello world dialog text", "queries": []}]
# Explicit query-aware type behaves exactly like the default.
calls.clear()
ctx = RuntimeContext(
query="alpha",
limit=5,
_search={"_compress": {"session": "true"}, "queries": ["orig question"], "type": "query-aware"},
)
await step(ctx)
assert calls == [
{"name": "compressor", "text": "hello world dialog text", "queries": ["orig question", "alpha"]},
]
asyncio.run(run())
def test_search_v2_step_keeps_original_body_when_compressed_is_longer():
"""A compressed body longer than the original is rejected in favor of the original."""
async def run():
session = _chunk("s1", "session/dialog/s1.jsonl", "short", "vector", 0.9)
store = FakeSearchStore(vector_results=[session])
step = SearchV2Step(file_store=store, expand_links=False)
async def fake_run_job(name, /, **kwargs): # pylint: disable=unused-argument
return Response(success=True, answer="a much longer compressed output")
step.run_job = fake_run_job
ctx = RuntimeContext(query="alpha", _search={"_compress": {"session": True}})
resp = await step(ctx)
assert "short" in resp.answer
assert "much longer" not in resp.answer
asyncio.run(run())
def test_search_v2_step_compression_strips_input_and_adds_marker():
"""The compressor receives the stripped rendered body; adopted output is
prefixed with the ``compressed session chunk:`` marker only (no line numbers)."""
async def run():
session = FileChunk(
id="s1",
path="session/dialog/s1.jsonl",
text="first message line\nsecond message line",
start_line=4,
end_line=5,
scores={"vector": 0.9, "score": 0.9},
)
store = FakeSearchStore(vector_results=[session])
step = SearchV2Step(file_store=store, expand_links=False)
calls: list[dict] = []
async def fake_run_job(name, /, **kwargs):
calls.append({"name": name, **kwargs})
return Response(success=True, answer="first compact\nsecond compact\n")
step.run_job = fake_run_job
ctx = RuntimeContext(query="alpha", _search={"_compress": {"session": "true"}})
resp = await step(ctx)
assert calls[0]["text"] == "first message line\nsecond message line"
assert "compressed session chunk:\nfirst compact\nsecond compact" in resp.answer
asyncio.run(run())
def test_search_v2_step_compression_input_is_line_aligned_with_session_jsonl():
"""Serialized Msg jsonl lines render one message per line (newlines flattened),
so compressor input line ``i`` maps to session file line ``start_line + i``."""
async def run():
m1 = Msg(name="user", role="user", content=[{"type": "text", "text": "line one\nline two"}])
m2 = Msg(name="assistant", role="assistant", content=[{"type": "text", "text": "long answer"}])
session = FileChunk(
id="s1",
path="session/dialog/s1.jsonl",
text=m1.model_dump_json() + "\n" + m2.model_dump_json(),
start_line=7,
end_line=8,
scores={"vector": 0.9, "score": 0.9},
)
store = FakeSearchStore(vector_results=[session])
step = SearchV2Step(file_store=store, expand_links=False)
calls: list[dict] = []
async def fake_run_job(name, /, **kwargs):
calls.append({"name": name, **kwargs})
return Response(success=True, answer="[user @ t] u\n[assistant @ t] a")
step.run_job = fake_run_job
ctx = RuntimeContext(query="alpha", _search={"_compress": {"session": "true"}})
resp = await step(ctx)
sent_lines = calls[0]["text"].splitlines()
assert len(sent_lines) == 2
assert sent_lines[0].startswith("[user @") and "line one line two" in sent_lines[0]
assert sent_lines[1].startswith("[assistant @") and "long answer" in sent_lines[1]
assert "compressed session chunk:\n[user @ t] u\n[assistant @ t] a" in resp.answer
asyncio.run(run())
def test_search_v2_step_adopts_shorter_output_regardless_of_line_count():
"""A shorter compressor output is adopted even when its line count differs."""
async def run():
session = FileChunk(
id="s1",
path="session/dialog/s1.jsonl",
text="first message line\nsecond message line",
start_line=4,
end_line=5,
scores={"vector": 0.9, "score": 0.9},
)
store = FakeSearchStore(vector_results=[session])
step = SearchV2Step(file_store=store, expand_links=False)
async def fake_run_job(name, /, **kwargs): # pylint: disable=unused-argument
return Response(success=True, answer="merged into one line")
step.run_job = fake_run_job
ctx = RuntimeContext(query="alpha", _search={"_compress": {"session": "true"}})
resp = await step(ctx)
assert "compressed session chunk:\nmerged into one line" in resp.answer
assert "first message line" not in resp.answer
asyncio.run(run())
def test_search_v2_step_skips_compression_without_injected_flag():
"""Without the injected ``_search`` payload the compressor job is never invoked."""
async def run():
session = _chunk("s1", "session/dialog/s1.jsonl", "dialog text", "vector", 0.9)
store = FakeSearchStore(vector_results=[session])
step = SearchV2Step(file_store=store, expand_links=False)
async def fake_run_job(name, /, **kwargs): # pylint: disable=unused-argument
raise AssertionError("compressor job must not be called")
step.run_job = fake_run_job
ctx = RuntimeContext(query="alpha")
resp = await step(ctx)
assert "dialog text" in resp.answer
asyncio.run(run())
def test_search_v2_step_keeps_original_body_when_compression_fails():
"""A failed compressor response (success=False) never replaces the body."""
async def run():
session = _chunk("s1", "session/dialog/s1.jsonl", "dialog text", "vector", 0.9)
store = FakeSearchStore(vector_results=[session])
step = SearchV2Step(file_store=store, expand_links=False)
async def fake_run_job(name, /, **kwargs): # pylint: disable=unused-argument
return Response(success=False, answer="boom")
step.run_job = fake_run_job
ctx = RuntimeContext(query="alpha", _search={"_compress": {"session": "true"}})
resp = await step(ctx)
assert "dialog text" in resp.answer
assert "boom" not in resp.answer
asyncio.run(run())
def test_search_v2_step_keeps_original_body_when_compression_raises():
"""A compressor job that raises (e.g. temporary LLM outage) keeps the original
body for that entry while other entries are still compressed; the search itself
never fails."""
async def run():
failing = _chunk("s1", "session/dialog/s1.jsonl", "dialog text one", "vector", 0.9, line=2)
ok = _chunk("s2", "session/dialog/s2.jsonl", "dialog text two", "vector", 0.8, line=1)
store = FakeSearchStore(vector_results=[failing, ok])
step = SearchV2Step(file_store=store, expand_links=False)
async def fake_run_job(_name, /, **kwargs):
if "one" in kwargs.get("text", ""):
raise RuntimeError("temporary LLM outage")
return Response(success=True, answer="compressed!")
step.run_job = fake_run_job
ctx = RuntimeContext(query="alpha", _search={"_compress": {"session": "true"}})
resp = await step(ctx)
assert resp.success is True
# The failing entry keeps its original body.
assert "dialog text one" in resp.answer
# The healthy entry is still compressed.
assert "compressed!" in resp.answer
assert "dialog text two" not in resp.answer
asyncio.run(run())

90
tests/unit/test_source_format_merge.py
View file

@ -1,60 +1,68 @@
"""Unit tests for session-chunk merging in ``format_chunks_answer``.
"""Unit tests for session-chunk merging via ``merge_session_chunk_intervals``.
Session chunks (``*.jsonl`` under the dialog dir) whose line ranges overlap,
contain one another, or are adjacent are merged into their union before
rendering. Bodies here are plain (non-``Msg``) text, so ``render_chunk_body``
falls back to the raw chunk text — letting these tests assert the union
content and line order directly.
contain one another, or are adjacent are merged into their union by the caller
(``merge_session_chunk_intervals``) before rendering with
``render_chunk_entries`` + ``join_chunk_entries``. Bodies here are plain
(non-``Msg``) text lines, which pass through stripped — letting these tests
assert the union content and line order directly.
"""
from reme.schema import FileChunk
from reme.steps.index._source_format import format_chunks_answer
from reme.steps.index._source_format import join_chunk_entries, merge_session_chunk_intervals, render_chunk_entries
_DIALOG_DIR = "session"
def _render(chunks: list[FileChunk], dialog_dir: str, **kwargs) -> str:
"""Merge session chunks, render entries, then join — mirroring the search steps."""
return join_chunk_entries(
render_chunk_entries(merge_session_chunk_intervals(chunks, dialog_dir), dialog_dir, **kwargs),
)
def _chunk(start: int, end: int, text: str, score: float = 1.0, path: str = "session/s1.jsonl") -> FileChunk:
return FileChunk(path=path, start_line=start, end_line=end, text=text, scores={"score": score})
def test_overlapping_session_chunks_merge_into_union_without_duplicates():
"""Overlapping ranges collapse to one passage; the shared line is shown once, in order."""
a = _chunk(1, 3, "L1\nL2\nL3\n")
b = _chunk(3, 5, "L3\nL4\nL5\n")
a = _chunk(1, 3, "m1\nm2\nm3\n")
b = _chunk(3, 5, "m3\nm4\nm5\n")
answer = format_chunks_answer([a, b], _DIALOG_DIR, include_source=False)
answer = _render([a, b], _DIALOG_DIR, include_source=False)
assert answer == "L1\nL2\nL3\nL4\nL5\n"
assert answer == "m1\nm2\nm3\nm4\nm5"
def test_contained_session_chunk_is_absorbed_by_the_larger_range():
"""When one range fully contains another, only the union (the larger) is shown."""
big = _chunk(1, 5, "L1\nL2\nL3\nL4\nL5\n")
small = _chunk(2, 4, "L2\nL3\nL4\n")
big = _chunk(1, 5, "m1\nm2\nm3\nm4\nm5\n")
small = _chunk(2, 4, "m2\nm3\nm4\n")
answer = format_chunks_answer([big, small], _DIALOG_DIR, include_source=False)
answer = _render([big, small], _DIALOG_DIR, include_source=False)
assert answer == "L1\nL2\nL3\nL4\nL5\n"
assert answer == "m1\nm2\nm3\nm4\nm5"
def test_adjacent_session_chunks_merge_end_plus_one_equals_next_start():
"""Gap-free consecutive chunks (prev.end + 1 == next.start) merge into one union."""
a = _chunk(1, 3, "L1\nL2\nL3\n")
b = _chunk(4, 6, "L4\nL5\nL6\n")
a = _chunk(1, 3, "m1\nm2\nm3\n")
b = _chunk(4, 6, "m4\nm5\nm6\n")
answer = format_chunks_answer([a, b], _DIALOG_DIR, include_source=False)
answer = _render([a, b], _DIALOG_DIR, include_source=False)
assert answer == "L1\nL2\nL3\nL4\nL5\nL6\n"
assert answer == "m1\nm2\nm3\nm4\nm5\nm6"
def test_session_chunks_with_a_gap_are_not_merged():
"""A missing line between ranges (start > end + 1) keeps the chunks separate."""
a = _chunk(1, 3, "L1\nL2\nL3\n")
b = _chunk(5, 6, "L5\nL6\n") # line 4 missing -> not adjacent
a = _chunk(1, 3, "m1\nm2\nm3\n")
b = _chunk(5, 6, "m5\nm6\n") # line 4 missing -> not adjacent
answer = format_chunks_answer([a, b], _DIALOG_DIR, include_source=False)
answer = _render([a, b], _DIALOG_DIR, include_source=False)
assert answer == "L1\nL2\nL3\n\n\nL5\nL6\n"
assert answer == "m1\nm2\nm3\n\nm5\nm6"
def test_session_chunks_from_different_files_are_not_merged():
@ -62,9 +70,9 @@ def test_session_chunks_from_different_files_are_not_merged():
a = _chunk(1, 3, "A1\nA2\nA3\n", path="session/s1.jsonl")
b = _chunk(2, 4, "B2\nB3\nB4\n", path="session/s2.jsonl")
answer = format_chunks_answer([a, b], _DIALOG_DIR, include_source=False)
answer = _render([a, b], _DIALOG_DIR, include_source=False)
assert answer == "A1\nA2\nA3\n\n\nB2\nB3\nB4\n"
assert answer == "A1\nA2\nA3\n\nB2\nB3\nB4"
def test_non_session_chunks_are_never_merged():
@ -72,28 +80,28 @@ def test_non_session_chunks_are_never_merged():
a = _chunk(1, 3, "M1\nM2\nM3\n", path="daily/a.md")
b = _chunk(2, 4, "M2\nM3\nM4\n", path="daily/a.md")
answer = format_chunks_answer([a, b], _DIALOG_DIR, include_source=False)
answer = _render([a, b], _DIALOG_DIR, include_source=False)
assert answer == "M1\nM2\nM3\n\n\nM2\nM3\nM4\n"
assert answer == "M1\nM2\nM3\n\nM2\nM3\nM4"
def test_merge_preserves_line_order_regardless_of_input_rank_order():
"""A later, higher-ranked chunk does not reorder union content; lines stay chronological."""
later = _chunk(3, 5, "L3\nL4\nL5\n", score=9.0)
earlier = _chunk(1, 3, "L1\nL2\nL3\n", score=1.0)
later = _chunk(3, 5, "m3\nm4\nm5\n", score=9.0)
earlier = _chunk(1, 3, "m1\nm2\nm3\n", score=1.0)
# Higher-scored later-range chunk is listed first (as a ranker would).
answer = format_chunks_answer([later, earlier], _DIALOG_DIR, include_source=False)
answer = _render([later, earlier], _DIALOG_DIR, include_source=False)
assert answer == "L1\nL2\nL3\nL4\nL5\n"
assert answer == "m1\nm2\nm3\nm4\nm5"
def test_merged_header_spans_the_union_range_and_keeps_best_score():
"""With source headers, the merged entry reports the union range and the top score."""
a = _chunk(1, 3, "L1\nL2\nL3\n", score=2.0)
b = _chunk(3, 5, "L3\nL4\nL5\n", score=7.0)
a = _chunk(1, 3, "m1\nm2\nm3\n", score=2.0)
b = _chunk(3, 5, "m3\nm4\nm5\n", score=7.0)
answer = format_chunks_answer([a, b], _DIALOG_DIR, include_source=True)
answer = _render([a, b], _DIALOG_DIR, include_source=True)
assert answer.count("==========") == 2 # exactly one header (open + close markers)
assert "session/s1.jsonl:1-5" in answer
@ -102,13 +110,13 @@ def test_merged_header_spans_the_union_range_and_keeps_best_score():
def test_separate_intervals_in_same_file_stay_separate():
"""Two disjoint interval clusters in one file yield two merged units, ordered by line."""
a = _chunk(1, 2, "L1\nL2\n")
b = _chunk(3, 4, "L3\nL4\n") # adjacent to a -> merges with a into 1-4
c = _chunk(10, 11, "L10\nL11\n") # far away -> separate
a = _chunk(1, 2, "m1\nm2\n")
b = _chunk(3, 4, "m3\nm4\n") # adjacent to a -> merges with a into 1-4
c = _chunk(10, 11, "m10\nm11\n") # far away -> separate
answer = format_chunks_answer([a, b, c], _DIALOG_DIR, include_source=False)
answer = _render([a, b, c], _DIALOG_DIR, include_source=False)
assert answer == "L1\nL2\nL3\nL4\n\n\nL10\nL11\n"
assert answer == "m1\nm2\nm3\nm4\n\nm10\nm11"
def test_same_file_units_stay_adjacent_and_sorted_even_when_interleaved_by_rank():
@ -120,11 +128,11 @@ def test_same_file_units_stay_adjacent_and_sorted_even_when_interleaved_by_rank(
s1_low = _chunk(1, 3, "S1a\nS1b\nS1c\n", score=1.0, path="session/s1.jsonl")
# Rank order (as a ranker would emit, by score desc): s1[10-12], s2[1-3], s1[1-3].
answer = format_chunks_answer([s1_high, s2_mid, s1_low], _DIALOG_DIR, include_source=False)
answer = _render([s1_high, s2_mid, s1_low], _DIALOG_DIR, include_source=False)
# s1's two units are adjacent and sorted by start_line (1-3 before 10-12),
# placed at s1's earliest rank (0), so the whole s1 block precedes s2.
assert answer == "S1a\nS1b\nS1c\n\n\nS1x\nS1y\nS1z\n\n\nS2a\nS2b\nS2c\n"
assert answer == "S1a\nS1b\nS1c\n\nS1x\nS1y\nS1z\n\nS2a\nS2b\nS2c"
def test_same_file_units_adjacency_with_source_headers():
@ -133,7 +141,7 @@ def test_same_file_units_adjacency_with_source_headers():
s2_mid = _chunk(1, 3, "S2a\nS2b\nS2c\n", score=5.0, path="session/s2.jsonl")
s1_low = _chunk(1, 3, "S1a\nS1b\nS1c\n", score=1.0, path="session/s1.jsonl")
answer = format_chunks_answer([s1_high, s2_mid, s1_low], _DIALOG_DIR, include_source=True)
answer = _render([s1_high, s2_mid, s1_low], _DIALOG_DIR, include_source=True)
headers = [line for line in answer.splitlines() if line.startswith("==========")]
assert headers[0].split(" [")[0] == "========== session/s1.jsonl:1-3"

133
tests/unit/test_workspace_web_steps.py Normal file
View file

@ -0,0 +1,133 @@
"""Focused contracts used by the local ReMe workspace web client."""
import asyncio
import os
from datetime import datetime
from reme.components.agent_wrapper import BaseAgentWrapper
from reme.components.application_context import ApplicationContext
from reme.components.runtime_context import RuntimeContext
from reme.enumeration import ChunkEnum
from reme.schema import StreamChunk
from reme.steps.common.chat import ChatStep
from reme.steps.file_io.load import LoadStep
from reme.steps.file_io.save import SaveStep
class _StreamingAgent(BaseAgentWrapper):
"""Minimal agent that records resume/tool arguments and emits rich chunks."""
def __init__(self):
super().__init__()
self.reply_kwargs = {}
async def reply(self, inputs, **kwargs) -> dict:
raise AssertionError("ChatStep must always use reply_stream()")
async def reply_stream(self, inputs, **kwargs):
self.reply_kwargs = kwargs
yield StreamChunk(chunk_type=ChunkEnum.TOOL_CALL, chunk="{}", tool_call_id="tool-1", tool_call_name="search")
yield StreamChunk(chunk_type=ChunkEnum.CONTENT, chunk="hello", session_id="session-new")
def test_save_step_preserves_complete_markdown(tmp_path):
"""The editor save endpoint writes frontmatter and body verbatim."""
async def run():
target = tmp_path / "daily" / "note.md"
target.parent.mkdir(parents=True)
target.write_text("old", encoding="utf-8")
expected = datetime.fromtimestamp(target.stat().st_mtime).isoformat()
content = "---\nname: note\ncustom: keep\n---\n\n# Updated\n"
response = await SaveStep(app_context=ApplicationContext(workspace_dir=str(tmp_path)))(
path="daily/note.md",
content=content,
expected_mtime=expected,
)
assert response.success is True
assert target.read_text(encoding="utf-8") == content
assert response.metadata["path"] == "daily/note.md"
assert response.metadata["mtime"]
asyncio.run(run())
def test_load_step_returns_complete_content_and_mtime(tmp_path):
"""Editor loads are complete and include the concurrency token for save."""
async def run():
content = "# Large enough\n" + ("memory line\n" * 5000)
target = tmp_path / "note.md"
target.write_text(content, encoding="utf-8")
response = await LoadStep(app_context=ApplicationContext(workspace_dir=str(tmp_path)))(path="note.md")
assert response.success is True
assert response.answer == content
assert response.metadata["size"] == target.stat().st_size
assert response.metadata["mtime"]
asyncio.run(run())
def test_load_step_rejects_file_larger_than_editor_limit(tmp_path):
"""Oversized files fail explicitly instead of returning editable truncation."""
async def run():
(tmp_path / "note.md").write_text("0123456789", encoding="utf-8")
response = await LoadStep(app_context=ApplicationContext(workspace_dir=str(tmp_path)))(
path="note.md",
max_bytes=5,
)
assert response.success is False
assert response.metadata["code"] == "file_too_large"
asyncio.run(run())
def test_save_step_rejects_external_change(tmp_path):
"""An obsolete mtime cannot overwrite a file changed by another editor."""
async def run():
target = tmp_path / "note.md"
target.write_text("first", encoding="utf-8")
expected = datetime.fromtimestamp(target.stat().st_mtime).isoformat()
target.write_text("external", encoding="utf-8")
newer = target.stat().st_mtime + 2
os.utime(target, (newer, newer))
response = await SaveStep(app_context=ApplicationContext(workspace_dir=str(tmp_path)))(
path="note.md",
content="browser edit",
expected_mtime=expected,
)
assert response.success is False
assert response.metadata["code"] == "file_conflict"
assert target.read_text(encoding="utf-8") == "external"
asyncio.run(run())
def test_chat_step_streams_rich_chunks_and_resumes_session():
"""Web chat keeps StreamChunk metadata and exposes only read-only tools."""
async def run():
agent = _StreamingAgent()
queue = asyncio.Queue()
context = RuntimeContext(stream_queue=queue, query="hello", session_id="session-old")
response = await ChatStep(agent_wrapper=agent)(context)
chunks = [await queue.get(), await queue.get()]
assert response.answer == "hello"
assert chunks[0].tool_call_name == "search"
assert chunks[1].session_id == "session-new"
assert agent.reply_kwargs["resume"] == "session-old"
assert agent.reply_kwargs["job_tools"] == ["search", "list", "read", "stat", "traverse"]
asyncio.run(run())