diff --git a/.gitignore b/.gitignore index 6e5e64c3..725c9c0b 100644 --- a/.gitignore +++ b/.gitignore @@ -61,17 +61,14 @@ datasets/ # Claude Code skills (local only) .claude/skills/ -# Memory workspaces (keep dirs, ignore contents) -benchmark/memory_workspaces/ +# Benchmark memory workspaces (created on demand by run.py via mkdir) +benchmark/*/workspaces/ -# LongMemEval raw data (download via download.py) -benchmark/datasets/longmemeval/longmemeval_s_reme_cleaned.json +# Benchmark datasets (LongMemEval via download.py, BEAM via git clone) +benchmark/*/dataset/ -# BEAM dataset (clone via git) -benchmark/datasets/BEAM/ - -# Benchmark outputs -benchmark/results/ +# Benchmark outputs (created on demand by run.py via mkdir) +benchmark/*/results/ # integration tests outputs tests/integration/logs/ diff --git a/README.md b/README.md index 1c909fc8..532c6f59 100644 --- a/README.md +++ b/README.md @@ -59,6 +59,8 @@ memory, then continuously indexes, links, and consolidates that memory for futur ## 📰 News +- [2026.08] - [Experience-driven enhancement method](benchmark/toolmemory/README.md) of agent tool-use + execution built on ReMe is available on [arXiv:2608.03403](https://arxiv.org/abs/2608.03403). - [2026.07] - Introduced optional Cookbooks: [Daily Paper](cookbook/daily_paper/README.md) for paper discovery and analysis, and [Auto Fin](cookbook/auto-fin/README.md) for file-native ETF event research based on CLS news and historical market reactions. diff --git a/README_ZH.md b/README_ZH.md index 79ecf6f3..8dd1246c 100644 --- a/README_ZH.md +++ b/README_ZH.md @@ -52,6 +52,9 @@ Agent 能够可靠召回。 ## 📰 新闻 +- [2026.08] - 基于 ReMe 的智能体工具使用 + [经验驱动增强方法](benchmark/toolmemory/README_ZH.md)已发布，见 + [arXiv:2608.03403](https://arxiv.org/abs/2608.03403)。 - [2026.07] - 新增可选 Cookbook 工作流：[每日论文](cookbook/daily_paper/README_ZH.md)用于论文发现与解析， [Auto Fin](cookbook/auto-fin/README_ZH.md)用于结合财联社新闻和历史行情开展文件化 ETF 事件研究。 - [2026.07] - diff --git a/benchmark/README.md b/benchmark/README.md index bbbc101a..0522b09e 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -34,22 +34,32 @@ Model names and component wiring live in the ReMe configs referenced by each ben ## 2. Download Datasets -See [`datasets/README_EN.md`](datasets/README_EN.md) for full details. +Each benchmark keeps its own data under its directory: +`benchmark//dataset` (input data), `benchmark//workspaces` +(per-item memory workspaces), and `benchmark//results` (evaluation outputs). +All three are excluded from Git. -**LongMemEval** (downloaded from a HuggingFace mirror): +**LongMemEval** — ReMe uses only the **cleaned-S** split, hosted on HuggingFace: +[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2) +(the script downloads via the hf-mirror.com mirror; to use a different mirror, +modify `BASE_URL` in `download.py`): ```bash -cd benchmark/datasets/longmemeval -python download.py # downloads the cleaned-S dataset; skips if already present +cd benchmark/longmemeval +python download.py # saves dataset/longmemeval_s_reme_cleaned.json; skips if already present ``` -**BEAM** (public repository, cloned into `benchmark/datasets/`): +**BEAM** (public repository, cloned into `benchmark/beam/dataset/`): ```bash -cd benchmark/datasets +mkdir -p benchmark/beam/dataset +cd benchmark/beam/dataset git clone https://github.com/mohammadtavakoli78/BEAM.git ``` +After cloning, `benchmark/beam/dataset/BEAM/` should contain `chats/`, `src/`, +`topics/` and other subdirectories. + ## 3. Run LongMemEval From the repository root: @@ -78,12 +88,12 @@ python benchmark/longmemeval/run.py --eval_only # reuse existing w | `dataset.path` | Dataset file to evaluate (e.g. `longmemeval_s_reme_cleaned.json`); ground truth is included. | | `dataset.start_index` / `num_items` | Slice of items to evaluate. | | `dataset.question_types` | Filter by question type; empty = all. | -| `dataset.workspace_root` | Per-item workspace root (`benchmark/memory_workspaces/longmemeval-s`). | +| `dataset.workspace_root` | Per-item workspace root (`benchmark/longmemeval/workspaces/longmemeval-s`). | | `evaluation.num_workers` | `0` = auto (cpu-2), `1` = sequential, `>1` = parallel. | | `evaluation.filter_future_sessions` | Only ingest sessions with timestamp ≤ `question_date`. | | `reme.config` | ReMe config used (`lme.yaml`). | | `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | Dream triggering behavior. | -| `output.dir` | Results directory (`benchmark/results/longmemeval`). | +| `output.dir` | Results directory (`benchmark/longmemeval/results`). | ## 4. Run BEAM @@ -107,14 +117,14 @@ python benchmark/beam/run.py --eval_only # reuse existing workspac | Key | Meaning | | --- | --- | -| `dataset.beam_root` | BEAM dataset root (`benchmark/datasets/BEAM`). | +| `dataset.beam_root` | BEAM dataset root (`benchmark/beam/dataset/BEAM`). | | `dataset.chat_size` | Variant to run: `100K` / `500K` / `1M` / `10M`. | | `dataset.case_ids` | Specific cases (e.g. `["1","2"]`); empty = all cases. | | `dataset.start_index` / `num_items` | Case pagination (`num_items` `0` = all). | -| `dataset.workspace_root` | Per-case workspace root (`benchmark/memory_workspaces/beam`). | +| `dataset.workspace_root` | Per-case workspace root (`benchmark/beam/workspaces/beam`). | | `evaluation.num_workers` | `0` = auto, `1` = sequential, `>1` = parallel. | | `reme.config` | ReMe config used (`beam.yaml`). | -| `output.dir` | Results directory (`benchmark/results/beam`). | +| `output.dir` | Results directory (`benchmark/beam/results`). | ## 5. Outputs & Logs @@ -141,5 +151,5 @@ The script gracefully sends `SIGTERM` to the whole process tree, then escalates Recorded evaluation results are available in: -- [`result-longmemeval.md`](./result-longmemeval.md) -- [`result-beam.md`](./result-beam.md) +- [`longmemeval.md`](./results_md/longmemeval.md) +- [`beam.md`](./results_md/beam.md) diff --git a/benchmark/README_ZH.md b/benchmark/README_ZH.md index c384a01c..0ac8fd3e 100644 --- a/benchmark/README_ZH.md +++ b/benchmark/README_ZH.md @@ -32,22 +32,29 @@ EMBEDDING_BASE_URL=... ## 2. 下载数据集 -完整说明见 [`datasets/README.md`](datasets/README.md)。 +每个基准的数据都存放在各自目录下：`benchmark//dataset`（输入数据）、 +`benchmark//workspaces`（按条目隔离的记忆工作区）、 +`benchmark//results`（评测输出）。三者均不纳入 Git 版本管理。 -**LongMemEval**（从 HuggingFace 镜像下载）： +**LongMemEval** —— ReMe 仅使用 **cleaned-S** 版本，数据托管在 HuggingFace： +[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2) +（下载脚本经 hf-mirror.com 镜像源获取，如需更换源请修改 `download.py` 中的 `BASE_URL`）： ```bash -cd benchmark/datasets/longmemeval -python download.py # 下载 cleaned-S 数据文件，已存在则自动跳过 +cd benchmark/longmemeval +python download.py # 保存为 dataset/longmemeval_s_reme_cleaned.json，已存在则自动跳过 ``` -**BEAM**（公开仓库，clone 到 `benchmark/datasets/` 下）： +**BEAM**（公开仓库，clone 到 `benchmark/beam/dataset/` 下）： ```bash -cd benchmark/datasets +mkdir -p benchmark/beam/dataset +cd benchmark/beam/dataset git clone https://github.com/mohammadtavakoli78/BEAM.git ``` +clone 完成后，`benchmark/beam/dataset/BEAM/` 目录下应包含 `chats/`、`src/`、`topics/` 等子目录。 + ## 3. 运行 LongMemEval 在仓库根目录执行： @@ -76,12 +83,12 @@ python benchmark/longmemeval/run.py --eval_only # 复用已有工 | `dataset.path` | 待评测的数据集文件（如 `longmemeval_s_reme_cleaned.json`），已包含 ground truth。 | | `dataset.start_index` / `num_items` | 评测条目的切片范围。 | | `dataset.question_types` | 按问题类型过滤，空表示全部。 | -| `dataset.workspace_root` | 条目工作区根目录（`benchmark/memory_workspaces/longmemeval-s`）。 | +| `dataset.workspace_root` | 条目工作区根目录（`benchmark/longmemeval/workspaces/longmemeval-s`）。 | | `evaluation.num_workers` | `0` = 自动（cpu-2），`1` = 串行，`>1` = 并行。 | | `evaluation.filter_future_sessions` | 仅摄入时间戳 ≤ `question_date` 的会话。 | | `reme.config` | 使用的 ReMe 配置（`lme.yaml`）。 | | `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | dream 触发行为。 | -| `output.dir` | 结果目录（`benchmark/results/longmemeval`）。 | +| `output.dir` | 结果目录（`benchmark/longmemeval/results`）。 | ## 4. 运行 BEAM @@ -105,14 +112,14 @@ python benchmark/beam/run.py --eval_only # 复用已有工作区 | 配置项 | 含义 | | --- | --- | -| `dataset.beam_root` | BEAM 数据集根目录（`benchmark/datasets/BEAM`）。 | +| `dataset.beam_root` | BEAM 数据集根目录（`benchmark/beam/dataset/BEAM`）。 | | `dataset.chat_size` | 运行的变体：`100K` / `500K` / `1M` / `10M`。 | | `dataset.case_ids` | 指定 case（如 `["1","2"]`），空表示全部。 | | `dataset.start_index` / `num_items` | case 分页（`num_items` 为 `0` 表示全部）。 | -| `dataset.workspace_root` | case 工作区根目录（`benchmark/memory_workspaces/beam`）。 | +| `dataset.workspace_root` | case 工作区根目录（`benchmark/beam/workspaces/beam`）。 | | `evaluation.num_workers` | `0` = 自动，`1` = 串行，`>1` = 并行。 | | `reme.config` | 使用的 ReMe 配置（`beam.yaml`）。 | -| `output.dir` | 结果目录（`benchmark/results/beam`）。 | +| `output.dir` | 结果目录（`benchmark/beam/results`）。 | ## 5. 输出与日志 @@ -138,5 +145,5 @@ bash benchmark/kill.sh 已记录的评测结果见： -- [`result-longmemeval.md`](./result-longmemeval.md) -- [`result-beam.md`](./result-beam.md) +- [`longmemeval.md`](./results_md/longmemeval.md) +- [`beam.md`](./results_md/beam.md) diff --git a/benchmark/beam/config.yaml b/benchmark/beam/config.yaml index 32e7620b..423697a5 100644 --- a/benchmark/beam/config.yaml +++ b/benchmark/beam/config.yaml @@ -2,21 +2,22 @@ # This file controls what/how to evaluate. dataset: - beam_root: "benchmark/datasets/BEAM" # BEAM dataset root - chat_size: "1M" # 100K | 500K | 1M | 10M (dataset variant) + beam_root: "benchmark/beam/dataset/BEAM" # BEAM dataset root + chat_size: "1M" # 100K | 500K | 1M | 10M (dataset variant) case_ids: [] # empty = all cases; or ["1", "2", "3"] start_index: 0 # first case index (for pagination) num_items: 0 # 0 = all cases; >0 = limit - workspace_root: "benchmark/memory_workspaces/beam" # workspace root for case workspaces + workspace_root: "benchmark/beam/workspaces/beam" # workspace root for case workspaces evaluation: - num_workers: 18 # 0 = auto; 1 = sequential; >1 = parallel (per-case) + num_workers: 20 # 0 = auto; 1 = sequential; >1 = parallel (per-case) + compress_session: false # true = compress session chunks in search_v2 (query-aware); false = no compression reme: config: "beam.yaml" # reme config (in reme/config/) output: - dir: "benchmark/results/beam" + dir: "benchmark/beam/results" log_dir: "logs" # log directory (relative to project root) log_prefix: "beam" # benchmark name used in log filenames log_to_console: true diff --git a/benchmark/beam/run.py b/benchmark/beam/run.py index 97c620e5..06e35b7d 100644 --- a/benchmark/beam/run.py +++ b/benchmark/beam/run.py @@ -33,7 +33,7 @@ _PROJECT_ROOT = Path(__file__).parent.parent.parent load_dotenv(_PROJECT_ROOT / ".env") # Workspace root — read from config.yaml (dataset.workspace_root) -_WORKSPACE_ROOT_DEFAULT = "benchmark/memory_workspaces/beam" +_WORKSPACE_ROOT_DEFAULT = "benchmark/beam/workspaces/beam" # --------------------------------------------------------------------------- # Logging @@ -247,7 +247,7 @@ def get_available_cases(beam_root: Path, chat_size: str) -> list[str]: # --------------------------------------------------------------------------- # Answer generation # --------------------------------------------------------------------------- -async def answer_question_agentic(app, question: str) -> tuple[str, dict]: +async def answer_question_agentic(app, question: str, compress_session: bool = False) -> tuple[str, dict]: """Answer a probing question using ReMe's agentic_answer job. Returns (answer, metadata) @@ -264,6 +264,7 @@ async def answer_question_agentic(app, question: str) -> tuple[str, dict]: query_resp = await app.run_job( "agentic_answer", query=question, + compress_session=compress_session, ) answer = (query_resp.answer or "").strip() @@ -323,7 +324,8 @@ async def evaluate_case(eval_config: dict, case_id: str, eval_only: bool = False dataset_cfg = eval_config["dataset"] chat_size = dataset_cfg["chat_size"] - beam_root = _PROJECT_ROOT / dataset_cfg.get("beam_root", "benchmark/datasets/BEAM") + compress_session = bool(eval_config["evaluation"].get("compress_session", False)) + beam_root = _PROJECT_ROOT / dataset_cfg.get("beam_root", "benchmark/beam/dataset/BEAM") chat_path = beam_root / "chats" / chat_size / case_id / "chat.json" probing_questions_path = beam_root / "chats" / chat_size / case_id / "probing_questions" / "probing_questions.json" @@ -384,13 +386,20 @@ async def evaluate_case(eval_config: dict, case_id: str, eval_only: bool = False app = Application(**cfg) await app.start() + from reme.utils.evaluation_interface import check_agent_token_usage # noqa: E402 + + _MEM_AGENT_NAMES = ("default", "bench") sessions_ingested = 0 + memory_token_usage: dict[str, dict[str, int | None]] = {} try: if not eval_only: - # ── Phase 1: Ingest sessions ────────────────────────────── + # ── Phase 1: Ingest sessions (with token tracking) ───────── sessions = load_beam_chat(chat_path, chat_size, case_id) logger.info(f"[Case {case_id}] Loaded {len(sessions)} sessions from chat.json") + # Snapshot token counters before memory construction + mem_token_start = {name: check_agent_token_usage(name, app.context) for name in _MEM_AGENT_NAMES} + for i, session in enumerate(sessions): logger.info( f"[Case {case_id}] Ingesting session {i+1}/{len(sessions)}: " @@ -417,6 +426,17 @@ async def evaluate_case(eval_config: dict, case_id: str, eval_only: bool = False await app.run_job("digest_update") logger.info(f"[Case {case_id}] Ingestion complete.") + # Compute memory construction token deltas + for name in _MEM_AGENT_NAMES: + end_usage = check_agent_token_usage(name, app.context) + delta: dict[str, int | None] = {} + for metric in _TOKEN_USAGE_METRICS: + current = end_usage[metric] + start = mem_token_start[name][metric] + delta[metric] = None if current is None else current - (start or 0) + memory_token_usage[name] = delta + logger.info(f"[Case {case_id}] Memory construction token usage: {memory_token_usage}") + # ── Phase 2: Answer + Judge probing questions ─────────────── with open(probing_questions_path, encoding="utf-8") as f: probing_questions = json.load(f) @@ -452,6 +472,7 @@ async def evaluate_case(eval_config: dict, case_id: str, eval_only: bool = False agentic_answer, agentic_meta = await answer_question_agentic( app, question, + compress_session=compress_session, ) except Exception as e: logger.error(f"[Case {case_id}] Agentic answer failed: {e}") @@ -494,6 +515,7 @@ async def evaluate_case(eval_config: dict, case_id: str, eval_only: bool = False "sessions_ingested": sessions_ingested, "total_questions": len(all_question_results), "questions": all_question_results, + "memory_token_usage": memory_token_usage, } @@ -561,7 +583,7 @@ def main( # pylint: disable=too-many-statements setup_logging(log_level, reme_log_level, log_dir=log_dir_abs) dataset_cfg = eval_config["dataset"] chat_size = dataset_cfg["chat_size"] - beam_root = _PROJECT_ROOT / dataset_cfg.get("beam_root", "benchmark/datasets/BEAM") + beam_root = _PROJECT_ROOT / dataset_cfg.get("beam_root", "benchmark/beam/dataset/BEAM") # Determine which cases to run case_ids = dataset_cfg.get("case_ids") or [] @@ -593,7 +615,7 @@ def main( # pylint: disable=too-many-statements logger.info(f"Using {num_workers} worker(s)") # Create output directory - output_dir = _PROJECT_ROOT / output_cfg.get("dir", "benchmark/results/beam") + output_dir = _PROJECT_ROOT / output_cfg.get("dir", "benchmark/beam/results") output_dir.mkdir(parents=True, exist_ok=True) # Create workspace root directory @@ -697,10 +719,14 @@ def main( # pylint: disable=too-many-statements all_binary_scores: list[float] = [] all_tool_call_totals: list[int] = [] all_token_usages: list[dict[str, int | None]] = [] + all_memory_token_usages: list[dict[str, dict[str, int | None]]] = [] for case_result in results: if "error" in case_result: continue + mem_usage = case_result.get("memory_token_usage", {}) + if mem_usage: + all_memory_token_usages.append(mem_usage) for q in case_result.get("questions", []): judgment = q.get("agentic_judgment", {}) score = judgment.get("llm_judge_score", 0.0) @@ -723,6 +749,26 @@ def main( # pylint: disable=too-many-statements all_tool_call_totals.append(sum(metadata.get("tool_counts", {}).values())) all_token_usages.append(metadata.get("token_usage", {})) + # Memory construction token usage summary + if all_memory_token_usages: + print("\n ── Memory Construction Token Usage ──") + for agent_name in ("default", "bench"): + for metric in _TOKEN_USAGE_METRICS: + values = [ + usage[agent_name][metric] + for usage in all_memory_token_usages + if usage.get(agent_name, {}).get(metric) is not None + ] + if values: + total = sum(values) + mean, std = _mean_and_std(values) + print( + f" {agent_name}/{metric}: total={total} mean={mean:.2f} std={std:.2f} ({len(values)} cases)", + ) + else: + print(f" {agent_name}/{metric}: unavailable") + print() + print("\n ── AGENTIC ──") if all_scores: for qtype in sorted(type_scores.keys()): @@ -757,7 +803,14 @@ def main( # pylint: disable=too-many-statements continue n_qs = case_result.get("total_questions", 0) n_sessions = case_result.get("sessions_ingested", 0) + mem_usage = case_result.get("memory_token_usage", {}) parts = [f"Case {case_id}: {n_sessions} sessions, {n_qs} questions"] + # Append memory construction total tokens if available + for agent_name in ("default", "bench"): + agent_usage = mem_usage.get(agent_name, {}) + total = agent_usage.get("total_tokens") + if total is not None: + parts.append(f"mem_{agent_name}_tokens={total}") questions = case_result.get("questions", []) scores = [q.get("agentic_judgment", {}).get("llm_judge_score", 0.0) for q in questions] if scores: diff --git a/benchmark/datasets/README.md b/benchmark/datasets/README.md deleted file mode 100644 index 3a772a9d..00000000 --- a/benchmark/datasets/README.md +++ /dev/null @@ -1,40 +0,0 @@ -# 数据集下载说明 - -本目录包含 ReMe 评测所需的数据集。部分数据集体积较大，不纳入 Git 版本管理，需要手动下载。 - -## LongMemEval（cleaned-S） - -ReMe 仅使用 LongMemEval 的 **cleaned-S** 版本，数据托管在 HuggingFace： -[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2) -（下载脚本经 hf-mirror.com 镜像源获取）。 - -按以下步骤下载： - -```bash -cd benchmark/datasets/longmemeval - -# 下载 cleaned-S 数据文件（已存在则自动跳过） -python download.py -``` - -下载完成后，目录下应包含以下文件： - -| 文件名 | 说明 | -| --- | --- | -| `longmemeval_s_reme_cleaned.json` | cleaned-S 数据集，已包含 ground truth 字段 | -| `download.py` | 下载脚本（已随仓库提供） | - -> **注意**：下载脚本使用 hf-mirror.com 镜像源，如需更换源请修改 `download.py` 中的 `BASE_URL`。 - -下载完成后即可参照 [`benchmark/README_ZH.md`](../README_ZH.md) 运行 LongMemEval 评测。 - -## BEAM - -BEAM 数据集为公开仓库，直接 clone 到 `benchmark/datasets/` 目录下即可： - -```bash -cd benchmark/datasets -git clone https://github.com/mohammadtavakoli78/BEAM.git -``` - -clone 完成后，`benchmark/datasets/BEAM/` 目录下应包含 `chats/`、`src/`、`topics/` 等子目录。 diff --git a/benchmark/datasets/README_EN.md b/benchmark/datasets/README_EN.md deleted file mode 100644 index 24a7bab0..00000000 --- a/benchmark/datasets/README_EN.md +++ /dev/null @@ -1,40 +0,0 @@ -# Dataset Download Guide - -This directory contains datasets required for ReMe evaluation. Some datasets are large and excluded from Git version control — they must be downloaded manually. - -## LongMemEval (cleaned-S) - -ReMe uses only the **cleaned-S** split of LongMemEval, hosted on HuggingFace: -[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2) -(the script downloads via the hf-mirror.com mirror). - -Download it with: - -```bash -cd benchmark/datasets/longmemeval - -# Download the cleaned-S data file (skipped automatically if it already exists) -python download.py -``` - -After downloading, the directory should contain: - -| File | Description | -| --- | --- | -| `longmemeval_s_reme_cleaned.json` | cleaned-S dataset with ground truth fields included | -| `download.py` | Download script (included in repo) | - -> **Note**: The download script uses hf-mirror.com by default. To use a different mirror, modify `BASE_URL` in `download.py`. - -Once the download completes, follow [`benchmark/README.md`](../README.md) to run the LongMemEval evaluation. - -## BEAM - -BEAM is a public repository. Clone it directly into the `benchmark/datasets/` directory: - -```bash -cd benchmark/datasets -git clone https://github.com/mohammadtavakoli78/BEAM.git -``` - -After cloning, `benchmark/datasets/BEAM/` should contain `chats/`, `src/`, `topics/` and other subdirectories. diff --git a/benchmark/longmemeval/config.yaml b/benchmark/longmemeval/config.yaml index ecc06f52..72ee7b75 100644 --- a/benchmark/longmemeval/config.yaml +++ b/benchmark/longmemeval/config.yaml @@ -2,12 +2,12 @@ # This file controls what/how to evaluate. dataset: - path: "benchmark/datasets/longmemeval/longmemeval_s_reme_cleaned.json" + path: "benchmark/longmemeval/dataset/longmemeval_s_reme_cleaned.json" start_index: 0 # first item index num_items: 500 # how many items to evaluate (starting from start_index) max_sessions: 0 # 0 = all sessions; >0 = limit sessions per item for testing question_types: [] # filter by question_type; empty list = no filtering (all types) - workspace_root: "benchmark/memory_workspaces/longmemeval-s" # workspace root for item workspaces + workspace_root: "benchmark/longmemeval/workspaces/longmemeval-s" # workspace root for item workspaces evaluation: # LLM-as-judge uses the 'judge' as_llm component defined in lme.yaml @@ -15,6 +15,7 @@ evaluation: # Judgment is always binary (yes/no) — defined in lme/llm_judge.yaml num_workers: 32 # 0 = auto (cpu_count - 2, min 1); 1 = sequential; >1 = parallel filter_future_sessions: true # true = only ingest sessions with timestamp <= question_date + compress_session: false # true = compress session chunks in search_v2 (query-aware); false = no compression reme: config: "lme.yaml" # reme config to use (in reme/config/) @@ -25,7 +26,7 @@ reme: dream_max_units: 5 output: - dir: "benchmark/results/longmemeval" + dir: "benchmark/longmemeval/results" log_dir: "logs" # log directory (relative to project root) log_prefix: "longmemeval" # benchmark name used in log filenames log_to_console: true diff --git a/benchmark/datasets/longmemeval/download.py b/benchmark/longmemeval/download.py similarity index 86% rename from benchmark/datasets/longmemeval/download.py rename to benchmark/longmemeval/download.py index d6754640..115bb989 100644 --- a/benchmark/datasets/longmemeval/download.py +++ b/benchmark/longmemeval/download.py @@ -3,8 +3,9 @@ Source: https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2 (downloaded via the hf-mirror.com mirror for reliability). -The file ``longmemeval_s_reme_cleaned.json`` is saved under this directory using the same -name as on the remote (``benchmark/longmemeval/config.yaml`` points to it). +The file ``longmemeval_s_reme_cleaned.json`` is saved under ``dataset/`` next to this +script using the same name as on the remote (``benchmark/longmemeval/config.yaml`` +points to it). Usage: python download.py # download cleaned-S (skip if it already exists) @@ -15,7 +16,7 @@ import sys import urllib.request BASE_URL = "https://hf-mirror.com/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2/resolve/main" -TARGET_DIR = os.path.dirname(os.path.abspath(__file__)) +TARGET_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "dataset") # Files to download (saved with the same name as on the remote). FILES = [ @@ -59,6 +60,7 @@ def _progress(block_num, block_size, total_size): if __name__ == "__main__": + os.makedirs(TARGET_DIR, exist_ok=True) print(f"Downloading LongMemEval cleaned-S dataset to: {TARGET_DIR}\n") for fname in FILES: download_file(fname) diff --git a/benchmark/longmemeval/run.py b/benchmark/longmemeval/run.py index baf386ae..9e48af43 100644 --- a/benchmark/longmemeval/run.py +++ b/benchmark/longmemeval/run.py @@ -32,7 +32,7 @@ _PROJECT_ROOT = Path(__file__).parent.parent.parent load_dotenv(_PROJECT_ROOT / ".env") # Workspace root for evaluation items — read from config.yaml (dataset.workspace_root) -_WORKSPACE_ROOT_DEFAULT = "benchmark/memory_workspaces/longmemeval-s" +_WORKSPACE_ROOT_DEFAULT = "benchmark/longmemeval/workspaces/longmemeval-s" # --------------------------------------------------------------------------- # Logging @@ -426,6 +426,7 @@ async def evaluate_item(item: dict, eval_config: dict, item_index: int, eval_onl # ── Phase 4: Ask question via agentic_answer job (ReAct agent) ── question = item["question"] + compress_session = bool(eval_config["evaluation"].get("compress_session", False)) question_date_raw = item.get("question_date", "") question_dt = parse_haystack_date(question_date_raw) if question_date_raw else None query_time = to_iso(question_dt) if question_dt else "" @@ -440,7 +441,12 @@ async def evaluate_item(item: dict, eval_config: dict, item_index: int, eval_onl app.context, ) as token_usages, ): - query_resp = await app.run_job("agentic_answer", query=question, query_time=query_time) + query_resp = await app.run_job( + "agentic_answer", + query=question, + query_time=query_time, + compress_session=compress_session, + ) agentic_tool_counts = tool_counts agentic_token_usage = token_usages["bench"] agentic_response = (query_resp.answer or "").strip() @@ -593,7 +599,7 @@ def main( logger.info(f"Using {num_workers} worker(s)") # Create output directory - output_dir = _PROJECT_ROOT / output_cfg.get("dir", "benchmark/results/longmemeval") + output_dir = _PROJECT_ROOT / output_cfg.get("dir", "benchmark/longmemeval/results") output_dir.mkdir(parents=True, exist_ok=True) # Create workspace root directory diff --git a/benchmark/result-beam.md b/benchmark/result-beam.md deleted file mode 100644 index 4ec5b71a..00000000 --- a/benchmark/result-beam.md +++ /dev/null @@ -1,36 +0,0 @@ -# beam result - -## longmemeval版本的prompt - -### 100K - - -| 题型 | Prompted(limit=15) | Agentic | -|---|---|---| -| abstention | 0.525 | 0.575 | -| contradiction_resolution | 0.100 | 0.384 | -| event_ordering | 0.403 | 0.465 | -| information_extraction | 0.618 | 0.884 | -| instruction_following | 0.481 | 0.719 | -| knowledge_update | 0.637 | 0.650 | -| multi_session_reasoning | 0.444 | 0.633 | -| preference_following | 0.706 | 0.829 | -| summarization | 0.423 | 0.617 | -| temporal_reasoning | 0.344 | 0.550 | -| **OVERALL** | **0.468** | **0.631** | - -### 1M - -| 题型 | Prompted(limit=15) | Prompted Binary | Agentic | Agentic Binary | -|---|---|---|---|---| -| abstention | 0.464 | 0.464 | 0.514 | 0.514 | -| contradiction_resolution | 0.079 | 0.068 | 0.373 | 0.339 | -| event_ordering | 0.455 | 0.334 | 0.547 | 0.450 | -| information_extraction | 0.653 | 0.589 | 0.818 | 0.764 | -| instruction_following | 0.541 | 0.524 | 0.765 | 0.745 | -| knowledge_update | 0.571 | 0.507 | 0.636 | 0.629 | -| multi_session_reasoning | 0.426 | 0.324 | 0.593 | 0.540 | -| preference_following | 0.718 | 0.676 | 0.838 | 0.824 | -| summarization | 0.516 | 0.303 | 0.661 | 0.478 | -| temporal_reasoning | 0.198 | 0.169 | 0.394 | 0.383 | -| **OVERALL** | **0.462** | **0.396** | **0.614** | **0.567** | \ No newline at end of file diff --git a/benchmark/results_md/beam.md b/benchmark/results_md/beam.md new file mode 100644 index 00000000..651690d7 --- /dev/null +++ b/benchmark/results_md/beam.md @@ -0,0 +1,54 @@ +# beam result + +## longmemeval版本的prompt + +### 100K + + +agentscope==2.0.4.post1, conda reme 环境, 20 并发, eval-only（复用已构建 memory） +(2026-08-05, 20 cases / 400 Qs, 总耗时 46.0 min) + +| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q | +|---|---|---|---|---|---|---| +| abstention | 0.550 | 0.550 | 96,031 | 1,070 | 97,101 | 4.58 | +| contradiction_resolution | 0.438 | 0.412 | 32,263 | 872 | 33,135 | 2.48 | +| event_ordering | 0.501 | 0.423 | 140,195 | 5,163 | 145,358 | 4.70 | +| information_extraction | 0.873 | 0.832 | 50,245 | 883 | 51,128 | 3.15 | +| instruction_following | 0.750 | 0.725 | 37,986 | 848 | 38,834 | 2.67 | +| knowledge_update | 0.688 | 0.675 | 31,198 | 651 | 31,849 | 2.27 | +| multi_session_reasoning | 0.626 | 0.584 | 85,038 | 4,563 | 89,601 | 4.28 | +| preference_following | 0.925 | 0.912 | 34,281 | 989 | 35,270 | 2.50 | +| summarization | 0.623 | 0.461 | 89,657 | 2,056 | 91,713 | 4.12 | +| temporal_reasoning | 0.637 | 0.625 | 34,563 | 1,049 | 35,612 | 2.52 | +| **OVERALL** | **0.661** | **0.620** | **63,146** | **1,814** | **64,960** | **3.33** | + +Memory Construction 平均 token 消耗（default agent, 20 cases 全量构建）: + +| Agent | input tok/case | output tok/case | total tok/case | +|---|---|---|---| +| default | 2,172,316 | 136,697 | 2,309,013 | + +### 1M + +agentscope==2.0.4.post1, conda reme 环境, 20 并发, 全量构建 memory +(2026-08-05, 35 cases / 700 Qs, 总耗时 459.2 min) + +| 题型 | Agentic | Binary | input tok/q | output tok/q | total tok/q | tool calls/q | +|---|---|---|---|---|---|---| +| abstention | 0.429 | 0.429 | 118,707 | 1,178 | 119,886 | 4.20 | +| contradiction_resolution | 0.391 | 0.364 | 49,787 | 810 | 50,597 | 2.50 | +| event_ordering | 0.558 | 0.456 | 201,514 | 3,889 | 205,403 | 4.79 | +| information_extraction | 0.809 | 0.772 | 78,950 | 894 | 79,844 | 3.00 | +| instruction_following | 0.852 | 0.832 | 55,757 | 924 | 56,681 | 2.81 | +| knowledge_update | 0.779 | 0.771 | 45,981 | 665 | 46,646 | 2.37 | +| multi_session_reasoning | 0.658 | 0.612 | 138,133 | 2,873 | 141,006 | 4.40 | +| preference_following | 0.798 | 0.777 | 51,796 | 920 | 52,716 | 2.53 | +| summarization | 0.693 | 0.537 | 158,794 | 2,905 | 161,700 | 4.44 | +| temporal_reasoning | 0.536 | 0.536 | 100,176 | 3,148 | 103,324 | 3.90 | +| **OVERALL** | **0.650** | **0.609** | **99,959** | **1,821** | **101,780** | **3.49** | + +Memory Construction 平均 token 消耗（default agent, 35 cases 全量构建）: + +| Agent | input tok/case | output tok/case | total tok/case | +|---|---|---|---| +| default | 31,943,817 | 1,417,061 | 33,360,878 | \ No newline at end of file diff --git a/benchmark/result-longmemeval.md b/benchmark/results_md/longmemeval.md similarity index 100% rename from benchmark/result-longmemeval.md rename to benchmark/results_md/longmemeval.md diff --git a/benchmark/toolmemory/README.md b/benchmark/toolmemory/README.md new file mode 100644 index 00000000..7f358018 --- /dev/null +++ b/benchmark/toolmemory/README.md @@ -0,0 +1,98 @@ +## Towards Robust Tool Use in Agents via Experience-Driven Adaptive Guidance + +**Language**: English (default) / [中文](./README_ZH.md) + +> Paper: [arXiv:2608.03403](https://arxiv.org/abs/2608.03403) +> Code: [https://github.com/WangCan1178/ExpG](https://github.com/WangCan1178/ExpG) + +

+ ExpG challenges and overview +

+ +### Overview + +This folder archives **ExpG**, a tool-use enhancement built on [Agentscope ReMe](https://github.com/agentscope-ai/ReMe). ExpG mines, distills, and reuses experience from historical tool calls to provide **capability boundaries** and **best-practice guidance**, which helps agents: + +- Select and invoke tools more robustly under dynamic or noisy environments; +- Let smaller models with guidance outperform larger, memoryless baselines; +- Improve consistently across tool selection, tool calling, and response generation. + +**How ReMe is used:** Start the Tool Memory service; historical tool calls are written and evaluated via `add_tool_call_result`, distilled into tool-level guidance via `summary_tool_memory`, then retrieved and injected into later reasoning via `retrieve_tool_memory`. ReMe provides the vector store and service APIs; the acquisition / distillation / reuse strategy is implemented by ExpG. Full implementation and experiments are in [WangCan1178/ExpG](https://github.com/WangCan1178/ExpG). + +--- + +### ExpG Mechanism + +ExpG treats tool invocations as learnable experience and runs a three-stage pipeline: + +1. **Experience Acquisition** + - Analyze invocation quality from historical trajectories (success/failure, cost, latency, etc.); + - Build structured experience units per tool, recording context, parameter patterns, and outcomes. + +2. **Experience Distillation** + - Filter noisy or unhelpful experiences and keep representative patterns; + - Aggregate by equivalence classes to cover common and rare failure modes; + - Summarize with an LLM into generalizable textual guidance. + +3. **Experience Reuse** + - Retrieve relevant experience / guidance for future tasks; + - Inject guidance into tool selection, argument generation, and response synthesis; + - Improve stability under dynamic environments and imperfect feedback. + +--- + +### Main Results + +Performance comparison (%) across MetaTool, API-Bank, and BFCL-V3. **Bold** indicates the best results within each model. + +| Model | Method | MetaTool Pass@1 | MetaTool Avg@3 | MetaTool Pass@3 | API-Bank Pass@1 | API-Bank Avg@3 | API-Bank Pass@3 | BFCL-V3 Pass@1 | BFCL-V3 Avg@3 | BFCL-V3 Pass@3 | Total Pass@1 | Total Avg@3 | Total Pass@3 | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| GPT-5 nano | No Method | 72.62 | 72.76 | 78.49 | 82.96 | 83.46 | 86.97 | 53.80 | 53.00 | 60.95 | 70.82 | 70.62 | 76.63 | +| GPT-5 nano | Few-shot | 74.12 | 75.11 | 82.32 | 83.71 | 83.96 | **87.22** | 56.18 | 55.24 | 61.39 | 72.36 | 72.65 | 79.28 | +| GPT-5 nano | DRAFT | 73.94 | 73.04 | 78.97 | 84.21 | 83.46 | **87.22** | 57.27 | 57.27 | 62.26 | 72.52 | 71.58 | 77.23 | +| GPT-5 nano | Mem0 | 74.96 | 76.13 | 82.92 | 84.96 | 85.21 | **87.22** | 60.95 | 61.61 | 65.08 | 73.98 | 74.67 | 80.35 | +| GPT-5 nano | **ExpG** | **81.67** | **82.07** | **84.60** | **86.72** | **86.55** | **87.22** | **64.43** | **63.99** | **66.38** | **79.32** | **79.22** | **81.69** | +| DeepSeek-V3 | No Method | 83.10 | 82.94 | 84.66 | 84.71 | 84.38 | 85.46 | 58.79 | 59.65 | 65.94 | 78.92 | 78.66 | 81.37 | +| DeepSeek-V3 | Few-shot | 82.74 | 83.90 | 86.28 | 85.21 | 84.63 | 86.22 | 60.52 | 60.30 | 67.90 | 79.08 | 79.45 | 82.92 | +| DeepSeek-V3 | DRAFT | 80.23 | 80.79 | 82.44 | 84.96 | 85.63 | 86.47 | 62.26 | 61.61 | 68.55 | 77.70 | 77.80 | 80.54 | +| DeepSeek-V3 | Mem0 | 83.88 | 84.56 | 86.40 | 85.46 | 85.55 | 86.47 | 65.08 | 65.15 | 68.33 | 80.70 | 80.91 | 83.12 | +| DeepSeek-V3 | **ExpG** | **85.26** | **85.38** | **86.52** | **87.72** | **87.39** | **87.97** | **69.41** | **69.92** | **72.02** | **82.76** | **82.61** | **84.11** | +| Qwen3-8B | No Method | 76.51 | 76.97 | 77.71 | 83.96 | 83.88 | 84.21 | 58.79 | 58.28 | 60.30 | 74.46 | 74.41 | 75.56 | +| Qwen3-8B | Few-shot | 79.93 | 79.83 | 82.92 | 83.71 | 82.62 | 84.96 | 60.09 | 59.29 | 61.39 | 76.91 | 76.27 | 79.32 | +| Qwen3-8B | DRAFT | 78.19 | 77.33 | 77.89 | 85.71 | 84.96 | 85.46 | 60.74 | 60.30 | 62.91 | 76.20 | 75.18 | 76.35 | +| Qwen3-8B | Mem0 | 75.07 | 75.47 | 82.38 | 86.22 | 86.05 | 86.47 | 63.34 | 64.93 | 66.16 | 74.69 | 74.98 | 80.07 | +| Qwen3-8B | **ExpG** | **83.52** | **84.88** | **85.08** | **86.47** | **87.89** | **87.97** | **67.46** | **66.96** | **68.33** | **81.06** | **81.82** | **82.48** | +| Qwen3-32B | No Method | 80.05 | 79.43 | 80.17 | 84.71 | 84.88 | 85.21 | 65.15 | 65.08 | 66.16 | 78.05 | 77.55 | 78.41 | +| Qwen3-32B | **ExpG** | **84.68** | **85.02** | **86.28** | **86.97** | **87.30** | **87.72** | **70.72** | **71.01** | **73.32** | **82.48** | **82.56** | **84.14** | +| Qwen3-235B | No Method | 78.25 | 79.23 | 80.29 | 85.46 | 85.46 | 85.71 | 71.37 | 71.15 | 73.54 | 78.13 | 78.49 | 79.91 | +| Qwen3-235B | **ExpG** | **86.34** | **86.70** | **86.94** | **87.47** | **86.97** | **88.22** | **79.61** | **78.52** | **80.04** | **85.29** | **84.98** | **85.69** | + +--- + +### Reference Code + +| Path | Role | +| --- | --- | +| [`tool_memory.py`](./tool_memory.py) | HTTP client for official ReMe Tool Memory APIs (`add_tool_call_result` / `summary_tool_memory` / `retrieve_tool_memory`) | +| [`parse_tool_call_result_prompt.yaml`](./parse_tool_call_result_prompt.yaml) | Prompt for multi-aspect evaluation of each tool call | +| [`summary_tool_memory_prompt.yaml`](./summary_tool_memory_prompt.yaml) | Prompt for summarizing tool call history into guidance | +| [`tool_memory_flows.yaml`](./tool_memory_flows.yaml) | Tool Memory flow / op config excerpt | + +These are reference snippets. For the full runnable codebase, see [WangCan1178/ExpG](https://github.com/WangCan1178/ExpG). + +--- + +### Citation + +```bibtex +@misc{wang2026expg, + title = {Towards Robust Tool Use in Agents via Experience-Driven Adaptive Guidance}, + author = {Can Wang and Haoran Chen and Li Yu and Ding Hao and Bohai Zhao and Zhaoyang Liu and Zhiying Tu}, + year = {2026}, + eprint = {2608.03403}, + archivePrefix = {arXiv}, + primaryClass = {cs.AI}, + url = {https://arxiv.org/abs/2608.03403}, + howpublished = {\url{https://github.com/WangCan1178/ExpG}} +} +``` diff --git a/benchmark/toolmemory/README_ZH.md b/benchmark/toolmemory/README_ZH.md new file mode 100644 index 00000000..b0c5b89b --- /dev/null +++ b/benchmark/toolmemory/README_ZH.md @@ -0,0 +1,98 @@ +## Towards Robust Tool Use in Agents via Experience-Driven Adaptive Guidance + +**语言**：中文 / [English](./README.md) + +> 论文：[arXiv:2608.03403](https://arxiv.org/abs/2608.03403) +> 代码：[https://github.com/WangCan1178/ExpG](https://github.com/WangCan1178/ExpG) + +

+ ExpG 挑战与概览 +

+ +### 简介 + +本目录归档基于 [Agentscope ReMe](https://github.com/agentscope-ai/ReMe) 的工具使用增强工作 **ExpG**：在 ReMe 记忆框架之上，从历史工具调用中挖掘、提炼并复用经验，为智能体提供工具的 **能力边界** 与 **最佳实践指导**，从而： + +- 在动态或有噪环境下更鲁棒地选择和调用工具； +- 让较小模型在带有经验指导时超越更大、但无记忆的基线； +- 在工具选择、工具调用和响应生成等多个阶段带来一致收益。 + +**如何使用 ReMe：** 启动 Tool Memory 服务后，历史工具调用经 `add_tool_call_result` 写入并评估，经 `summary_tool_memory` 蒸馏成工具级指导，再经 `retrieve_tool_memory` 取回并注入后续推理。向量存储与服务接口由 ReMe 提供，经验获取 / 蒸馏 / 复用策略由 ExpG 实现。完整实现与实验见 [WangCan1178/ExpG](https://github.com/WangCan1178/ExpG)。 + +--- + +### ExpG 机制概览 + +ExpG 将工具调用视为可学习经验，并通过三阶段流水线完成经验的获取、提炼与复用： + +1. **经验获取（Experience Acquisition）** + - 从历史工具调用轨迹中分析调用质量（成功/失败、代价、时间等）； + - 针对不同工具构建结构化的经验单元，记录调用上下文、参数模式和结果。 + +2. **经验蒸馏（Experience Distillation）** + - 过滤无效 / 噪声经验，保留具有代表性的调用模式； + - 基于“等价类”视角对经验进行聚合，覆盖常见模式与稀有失败模式； + - 使用 LLM 对经验进行总结，形成可泛化的文本化指导（guidance）。 + +3. **经验复用（Experience Reuse）** + - 在未来任务中，根据当前工具调用上下文检索相关经验 / 指导； + - 将经验引导融入到工具选择、参数生成和响应整理等环节； + - 使得代理在面对动态环境和不完美反馈时仍能保持稳定表现。 + +--- + +### 主实验结果 + +MetaTool、API-Bank、BFCL-V3 上的性能对比（%）。**加粗**为各模型组内最优。 + +| Model | Method | MetaTool Pass@1 | MetaTool Avg@3 | MetaTool Pass@3 | API-Bank Pass@1 | API-Bank Avg@3 | API-Bank Pass@3 | BFCL-V3 Pass@1 | BFCL-V3 Avg@3 | BFCL-V3 Pass@3 | Total Pass@1 | Total Avg@3 | Total Pass@3 | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| GPT-5 nano | No Method | 72.62 | 72.76 | 78.49 | 82.96 | 83.46 | 86.97 | 53.80 | 53.00 | 60.95 | 70.82 | 70.62 | 76.63 | +| GPT-5 nano | Few-shot | 74.12 | 75.11 | 82.32 | 83.71 | 83.96 | **87.22** | 56.18 | 55.24 | 61.39 | 72.36 | 72.65 | 79.28 | +| GPT-5 nano | DRAFT | 73.94 | 73.04 | 78.97 | 84.21 | 83.46 | **87.22** | 57.27 | 57.27 | 62.26 | 72.52 | 71.58 | 77.23 | +| GPT-5 nano | Mem0 | 74.96 | 76.13 | 82.92 | 84.96 | 85.21 | **87.22** | 60.95 | 61.61 | 65.08 | 73.98 | 74.67 | 80.35 | +| GPT-5 nano | **ExpG** | **81.67** | **82.07** | **84.60** | **86.72** | **86.55** | **87.22** | **64.43** | **63.99** | **66.38** | **79.32** | **79.22** | **81.69** | +| DeepSeek-V3 | No Method | 83.10 | 82.94 | 84.66 | 84.71 | 84.38 | 85.46 | 58.79 | 59.65 | 65.94 | 78.92 | 78.66 | 81.37 | +| DeepSeek-V3 | Few-shot | 82.74 | 83.90 | 86.28 | 85.21 | 84.63 | 86.22 | 60.52 | 60.30 | 67.90 | 79.08 | 79.45 | 82.92 | +| DeepSeek-V3 | DRAFT | 80.23 | 80.79 | 82.44 | 84.96 | 85.63 | 86.47 | 62.26 | 61.61 | 68.55 | 77.70 | 77.80 | 80.54 | +| DeepSeek-V3 | Mem0 | 83.88 | 84.56 | 86.40 | 85.46 | 85.55 | 86.47 | 65.08 | 65.15 | 68.33 | 80.70 | 80.91 | 83.12 | +| DeepSeek-V3 | **ExpG** | **85.26** | **85.38** | **86.52** | **87.72** | **87.39** | **87.97** | **69.41** | **69.92** | **72.02** | **82.76** | **82.61** | **84.11** | +| Qwen3-8B | No Method | 76.51 | 76.97 | 77.71 | 83.96 | 83.88 | 84.21 | 58.79 | 58.28 | 60.30 | 74.46 | 74.41 | 75.56 | +| Qwen3-8B | Few-shot | 79.93 | 79.83 | 82.92 | 83.71 | 82.62 | 84.96 | 60.09 | 59.29 | 61.39 | 76.91 | 76.27 | 79.32 | +| Qwen3-8B | DRAFT | 78.19 | 77.33 | 77.89 | 85.71 | 84.96 | 85.46 | 60.74 | 60.30 | 62.91 | 76.20 | 75.18 | 76.35 | +| Qwen3-8B | Mem0 | 75.07 | 75.47 | 82.38 | 86.22 | 86.05 | 86.47 | 63.34 | 64.93 | 66.16 | 74.69 | 74.98 | 80.07 | +| Qwen3-8B | **ExpG** | **83.52** | **84.88** | **85.08** | **86.47** | **87.89** | **87.97** | **67.46** | **66.96** | **68.33** | **81.06** | **81.82** | **82.48** | +| Qwen3-32B | No Method | 80.05 | 79.43 | 80.17 | 84.71 | 84.88 | 85.21 | 65.15 | 65.08 | 66.16 | 78.05 | 77.55 | 78.41 | +| Qwen3-32B | **ExpG** | **84.68** | **85.02** | **86.28** | **86.97** | **87.30** | **87.72** | **70.72** | **71.01** | **73.32** | **82.48** | **82.56** | **84.14** | +| Qwen3-235B | No Method | 78.25 | 79.23 | 80.29 | 85.46 | 85.46 | 85.71 | 71.37 | 71.15 | 73.54 | 78.13 | 78.49 | 79.91 | +| Qwen3-235B | **ExpG** | **86.34** | **86.70** | **86.94** | **87.47** | **86.97** | **88.22** | **79.61** | **78.52** | **80.04** | **85.29** | **84.98** | **85.69** | + +--- + +### 参考代码 + +| 路径 | 作用 | +| --- | --- | +| [`tool_memory.py`](./tool_memory.py) | 官方风格 ReMe Tool Memory HTTP 客户端（`add_tool_call_result` / `summary_tool_memory` / `retrieve_tool_memory`） | +| [`parse_tool_call_result_prompt.yaml`](./parse_tool_call_result_prompt.yaml) | 单次工具调用多维评估用的 prompt | +| [`summary_tool_memory_prompt.yaml`](./summary_tool_memory_prompt.yaml) | 将工具调用历史总结为 guidance 的 prompt | +| [`tool_memory_flows.yaml`](./tool_memory_flows.yaml) | Tool Memory 相关的 flow / op 配置摘录 | + +以上为参考片段。完整可运行代码见 [WangCan1178/ExpG](https://github.com/WangCan1178/ExpG)。 + +--- + +### 引用 + +```bibtex +@misc{wang2026expg, + title = {Towards Robust Tool Use in Agents via Experience-Driven Adaptive Guidance}, + author = {Can Wang and Haoran Chen and Li Yu and Ding Hao and Bohai Zhao and Zhaoyang Liu and Zhiying Tu}, + year = {2026}, + eprint = {2608.03403}, + archivePrefix = {arXiv}, + primaryClass = {cs.AI}, + url = {https://arxiv.org/abs/2608.03403}, + howpublished = {\url{https://github.com/WangCan1178/ExpG}} +} +``` diff --git a/benchmark/toolmemory/gitcha.png b/benchmark/toolmemory/gitcha.png new file mode 100644 index 00000000..bb1152ec Binary files /dev/null and b/benchmark/toolmemory/gitcha.png differ diff --git a/benchmark/toolmemory/parse_tool_call_result_prompt.yaml b/benchmark/toolmemory/parse_tool_call_result_prompt.yaml new file mode 100644 index 00000000..371a12aa --- /dev/null +++ b/benchmark/toolmemory/parse_tool_call_result_prompt.yaml @@ -0,0 +1,49 @@ +prompt: | + You are an expert in evaluating tool invocation process. The tool is invoked by an AI agent. + + Tool invocation Information: + - Tool Name: {tool_name} + - Success Flag: {success_flag} + - Time Cost: {time_cost}s + - Token Cost: {token_cost} tokens + - Agent Context: {context} + - Input Parameters: {input_params} + - Tool Response: {response} + - Tool Schema: {schema} + + Evaluation Method: + Start from a default score list of scores = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]. + For each item below that is satisfied, assign 1 point to the corresponding index. + The final scores should be a list of 10 integers, each being either 0 or 1. + + 1. Use Quality (total 2 points. If context is provided, use it as an aid when evaluating): + - Index 1: Should the tool be invoked now? Consider whether all necessary information for the tool's invocation is ready, and whether the tool execution environment is correct. If it is a multi-round conversation, also consider the dependency relationships of the tool chain. + - Index 2: If should, is the chosen tool appropriate? + + 2. Input Quality (total 4 points. When evaluating, consider both the context and the tool schema): + - Index 3: Are all required parameters provided? + - Index 4: Are the input parameters valid and supported by the tool? + - Index 5: Are the input parameters in the correct format for their respective fields? + - Index 6: Does the value (content) of input parameter correctly reflect and match the given context? + + 3. Response Quality (total 4 points): + - Index 7: Does the response provide meaningful and useful information? Or are there any error messages or information that can be used as guidance for agent invoking tool better? + - Index 8: Does the response match the tool's intended purpose/function? + - Index 9: Does the response value correct (content appropriate) given the input parameters? + - Index 10: Does the response help accomplish the task within the given context? + + Important: + 1. Sometimes there is not enough information in the context or schema to make a complete evaluation. In such cases, make your best judgment based on the available information. + 2. Some tools (commonly system tools such as mkdir, touch, echo, etc.) modify the external environment. Since these results cannot be obtained, they return "None" as the response. At this point, all the scores in the quality of the response should be obtained and should not be seen as a problem for the tool. + 3. Evaluation independently from the success flag. The success_flag indicates whether the tool executed without technical errors. The evaluation should evaluate the quality of the tool invocation. A tool can execute successfully (Success Flag=1) but still produce low-quality or irrelevant responses, leading to a low evaluation score. + 4. Sometimes an agent will execute multiple steps and invoke multiple tools to complete a task, but you only need to evaluate the use of one tool for one of the steps, not whether the final task is completed or not. + + Answer Format: + Please provide your answer in the following JSON format: + + ```json + { + "scores": [0,0,0,0,0,0,0,0,0,0], + "explanation": "A brief evaluation (2-3 sentences) explaining the quality of the tool invocation, based on your evaluation. Low-quality aspects need to be reified, especially the causes of tool invocation errors." + } + ``` diff --git a/benchmark/toolmemory/summary_tool_memory_prompt.yaml b/benchmark/toolmemory/summary_tool_memory_prompt.yaml new file mode 100644 index 00000000..bff805c1 --- /dev/null +++ b/benchmark/toolmemory/summary_tool_memory_prompt.yaml @@ -0,0 +1,32 @@ +prompt: | + You are an expert in analyzing tool usage patterns and generating practical usage guidance for agents. + + Tool Information: + - Tool Name: {tool_name} + - Tool Schema: {tool_schema} + + Recent Tool Invocation Experiences: + {experiences} + + Important: + 1. Assume the tool (tool schema) can't be changed, your task is to guide agent to use it better. + 2. Your answer must be based on the information given, don't make it up. If not enough data, state "Not enough data to determine Core Function/Success Patterns/Common Issues/Best Practices." + 3. Your answer will be used to guide the use of the tool in the future, so do not include content related to recent tool invocation experience such as "case #3" or "Call #2", but some values can be used as examples. + 4. Pay attention to information not mentioned in the tool schema, such as the response upon successful tool invocation. It's also welcome to uncover insights, such as how tools can be used more effectively, and possible dependencies between tools. But if they aren't, don't make them up. + 5. Finally, to avoid deriving incorrect guidance from individual invocation, check whether, if the agent follows the proposed guidance, it can perform better on all recent invocation histories. If not, revise the guidance until it can. Specifically: + - Don't write guidance in an absolute tone without a very deterministic message (meaning that all invocation histories are satisfied, otherwise it will result in failure). + - Sometimes there may be inconsistencies. Consider whether this is due to the context in which the tool is being used. + + Your Task: + Based on the tool invocation history, generate a concise and logical tool usage guidance following this structure: + 1. Core Function: What this tool does and when to use it. + 2. Success Patterns: Parameter patterns and usage scenarios that work well. + 3. Common Issues: Main pitfalls to avoid and why they fail. + 4. Best Practices: 2-3 actionable recommendations. + + Answer Format: + Provide a structured, concise guidance (max 200 words). Focus on actionable insights derived from actual usage data. Avoid generic advice and think step by step. + + ```txt + Your concise, data-driven tool usage guidance + ``` \ No newline at end of file diff --git a/benchmark/toolmemory/tool_memory.py b/benchmark/toolmemory/tool_memory.py new file mode 100644 index 00000000..4a433de6 --- /dev/null +++ b/benchmark/toolmemory/tool_memory.py @@ -0,0 +1,234 @@ +"""Official-style ReMe Tool Memory HTTP helpers. + +Aligned with ReMe Tool Memory HTTP APIs (see ReMe cookbook +``use_tool_memory_demo.py`` and docs under ``docs/tool_memory/``): + +- ``add_tool_call_result`` +- ``summary_tool_memory`` +- ``retrieve_tool_memory`` + +Response memories are read from ``metadata.memory_list[].content``. +This module does not use ExpG-only fields such as ``no_persist``, +``source_task``, or ``add_to``. +""" + +from __future__ import annotations + +import logging +from typing import Any, Dict, List, Optional + +import httpx + +logger = logging.getLogger(__name__) + +DEFAULT_BASE_URL = "http://localhost:8002" + + +class ToolMemoryFetcher: + """HTTP client for ReMe Tool Memory endpoints.""" + + def __init__( + self, + workspace_id: str, + base_url: str = DEFAULT_BASE_URL, + timeout: float = 60.0, + ) -> None: + self.workspace_id = workspace_id + self.base_url = base_url.rstrip("/") + self.timeout = timeout + + def _url(self, endpoint: str) -> str: + return f"{self.base_url}/{endpoint.lstrip('/')}" + + @staticmethod + def _join_tool_names(tool_names: List[str] | str) -> str: + if isinstance(tool_names, str): + return tool_names + return ",".join(tool_names) + + @staticmethod + def _memory_list(payload: Dict[str, Any]) -> List[Dict[str, Any]]: + metadata = payload.get("metadata") or {} + if not isinstance(metadata, dict): + return [] + memory_list = metadata.get("memory_list") or [] + return memory_list if isinstance(memory_list, list) else [] + + @classmethod + def _content_by_tool(cls, payload: Dict[str, Any]) -> Dict[str, str]: + result: Dict[str, str] = {} + for memory in cls._memory_list(payload): + if not isinstance(memory, dict): + continue + tool_name = str(memory.get("when_to_use") or "").strip() + content = memory.get("content") or "" + if tool_name: + result[tool_name] = str(content) + return result + + async def add_tool_call_result_async( + self, + tool_call_results: List[Dict[str, Any]], + ) -> Dict[str, Any]: + """Call ``add_tool_call_result``.""" + async with httpx.AsyncClient() as client: + response = await client.post( + self._url("add_tool_call_result"), + json={ + "workspace_id": self.workspace_id, + "tool_call_results": tool_call_results, + }, + timeout=self.timeout, + ) + response.raise_for_status() + return response.json() + + async def summary_tool_memory_async( + self, + tool_names: List[str] | str, + ) -> Dict[str, Any]: + """Call ``summary_tool_memory``.""" + async with httpx.AsyncClient() as client: + response = await client.post( + self._url("summary_tool_memory"), + json={ + "workspace_id": self.workspace_id, + "tool_names": self._join_tool_names(tool_names), + }, + timeout=self.timeout, + ) + response.raise_for_status() + return response.json() + + async def retrieve_tool_memory_async( + self, + tool_names: List[str] | str, + ) -> Dict[str, Any]: + """Call ``retrieve_tool_memory``.""" + async with httpx.AsyncClient() as client: + response = await client.post( + self._url("retrieve_tool_memory"), + json={ + "workspace_id": self.workspace_id, + "tool_names": self._join_tool_names(tool_names), + }, + timeout=self.timeout, + ) + response.raise_for_status() + return response.json() + + async def collect_memory_async( + self, + tool_names: List[str], + ) -> Dict[str, str]: + """Summarize then retrieve guidance for tools. + + Returns: + Mapping from tool name to memory ``content`` string. + """ + if not tool_names: + return {} + + names = self._join_tool_names(tool_names) + try: + summary = await self.summary_tool_memory_async(names) + if not summary.get("success"): + logger.warning("summary_tool_memory failed for %s", names) + except Exception as exc: # noqa: BLE001 + logger.warning("summary_tool_memory error for %s: %s", names, exc) + + try: + retrieved = await self.retrieve_tool_memory_async(names) + except Exception as exc: # noqa: BLE001 + logger.warning("retrieve_tool_memory error for %s: %s", names, exc) + return {} + + if not retrieved.get("success"): + logger.warning("retrieve_tool_memory failed for %s", names) + return {} + + return self._content_by_tool(retrieved) + + def add_tool_call_result( + self, + tool_call_results: List[Dict[str, Any]], + ) -> Dict[str, Any]: + """Sync wrapper for ``add_tool_call_result``.""" + with httpx.Client() as client: + response = client.post( + self._url("add_tool_call_result"), + json={ + "workspace_id": self.workspace_id, + "tool_call_results": tool_call_results, + }, + timeout=self.timeout, + ) + response.raise_for_status() + return response.json() + + def summary_tool_memory(self, tool_names: List[str] | str) -> Dict[str, Any]: + """Sync wrapper for ``summary_tool_memory``.""" + with httpx.Client() as client: + response = client.post( + self._url("summary_tool_memory"), + json={ + "workspace_id": self.workspace_id, + "tool_names": self._join_tool_names(tool_names), + }, + timeout=self.timeout, + ) + response.raise_for_status() + return response.json() + + def retrieve_tool_memory(self, tool_names: List[str] | str) -> Dict[str, Any]: + """Sync wrapper for ``retrieve_tool_memory``.""" + with httpx.Client() as client: + response = client.post( + self._url("retrieve_tool_memory"), + json={ + "workspace_id": self.workspace_id, + "tool_names": self._join_tool_names(tool_names), + }, + timeout=self.timeout, + ) + response.raise_for_status() + return response.json() + + def collect_memory(self, tool_names: List[str]) -> Dict[str, str]: + """Sync wrapper for summarize + retrieve. + + Prefer ``collect_memory_async`` inside an existing event loop. + """ + if not tool_names: + return {} + + names = self._join_tool_names(tool_names) + try: + summary = self.summary_tool_memory(names) + if not summary.get("success"): + logger.warning("summary_tool_memory failed for %s", names) + except Exception as exc: # noqa: BLE001 + logger.warning("summary_tool_memory error for %s: %s", names, exc) + + try: + retrieved = self.retrieve_tool_memory(names) + except Exception as exc: # noqa: BLE001 + logger.warning("retrieve_tool_memory error for %s: %s", names, exc) + return {} + + if not retrieved.get("success"): + logger.warning("retrieve_tool_memory failed for %s", names) + return {} + + return self._content_by_tool(retrieved) + + def get_memory_content( + self, + tool_names: List[str] | str, + ) -> Optional[str]: + """Retrieve and join memory contents for the given tools.""" + payload = self.retrieve_tool_memory(tool_names) + if not payload.get("success"): + return None + contents = [content for content in self._content_by_tool(payload).values() if content] + return "\n\n".join(contents) if contents else None diff --git a/benchmark/toolmemory/tool_memory_flows.yaml b/benchmark/toolmemory/tool_memory_flows.yaml new file mode 100644 index 00000000..b62d0571 --- /dev/null +++ b/benchmark/toolmemory/tool_memory_flows.yaml @@ -0,0 +1,45 @@ +# Tool Memory flow / op config excerpt used by ExpG. +# Full runnable code: https://github.com/WangCan1178/ExpG + +flow: + retrieve_tool_memory: + flow_content: retrieve_tool_memory_op + description: "Retrieves tool memories from the vector database based on tool names to provide tool usage patterns and best practices" + input_schema: + tool_names: + type: string + description: "Comma-separated tool names (e.g., 'tool_name1,tool_name2')" + required: true + + add_tool_call_result: + flow_content: parse_tool_call_result_op >> update_vector_store_op + description: "Evaluates and adds tool call results to the tool memory database, creating new memory or updating existing memory for the specified tool" + input_schema: + tool_call_results: + type: array + description: "List of tool call result objects, each containing: tool_name, input, output, success, time_cost, token_cost, create_time" + required: true + + summary_tool_memory: + flow_content: summary_tool_memory_op >> update_vector_store_op + description: "Analyzes tool call history and generates comprehensive usage patterns, best practices, and recommendations for the specified tools" + input_schema: + tool_names: + type: string + description: "Comma-separated tool names to summarize (e.g., 'tool_name1,tool_name2')" + required: true + +op: + parse_tool_call_result_op: + backend: parse_tool_call_result_op + llm: default + params: + max_history_tool_call_cnt: 100 + evaluation_sleep_interval: 1.0 + + summary_tool_memory_op: + backend: summary_tool_memory_op + llm: default + params: + data_from: '2025-09-10 10:56:58' + summary_sleep_interval: 1.0 diff --git a/reme/components/runtime_context.py b/reme/components/runtime_context.py index d9454900..355b94ce 100644 --- a/reme/components/runtime_context.py +++ b/reme/components/runtime_context.py @@ -73,6 +73,11 @@ class RuntimeContext: await self._enqueue(StreamChunk(chunk_type=chunk_type, chunk=chunk)) return self + async def add_stream_chunk(self, chunk: StreamChunk) -> "RuntimeContext": + """Emit a complete chunk while preserving session and tool metadata.""" + await self._enqueue(chunk) + return self + async def add_stream_done(self) -> "RuntimeContext": """Emit the terminal DONE marker to close the stream.""" await self._enqueue(StreamChunk(chunk_type=ChunkEnum.DONE, chunk="", done=True)) diff --git a/reme/config/beam.yaml b/reme/config/beam.yaml index 805f8769..430bb743 100644 --- a/reme/config/beam.yaml +++ b/reme/config/beam.yaml @@ -164,6 +164,33 @@ jobs: - messages steps: - backend: beam_auto_memory_step + # Long sessions are split into turn-aligned segments and fed to the + # agent incrementally; each segment holds at most this many words. + # ("segment" here is a slice of the message list, unrelated to file + # chunking in the index.) <= 0 disables splitting. + max_segment_words: 10000 + + # ── Text compression (direct LLM call, no agent) ── + compressor: + backend: base + description: "Compress text via a direct LLM call, optionally guided by queries as relevance filter" + parameters: + type: object + properties: + text: + type: string + description: "the text to compress" + queries: + type: array + description: "optional list of queries; content potentially relevant to any query is kept, content certainly irrelevant to all queries may be dropped" + items: + type: string + default: [] + required: + - text + steps: + - backend: compressor_step + as_llm: compressor # ── Reindex (full rebuild) ── reindex: @@ -218,7 +245,7 @@ jobs: - backend: search_v2_step vector_weight: 0.7 candidate_multiplier: 5.0 - expand_links: true + expand_links: false max_links_per_direction: 10 add_draft: @@ -511,6 +538,19 @@ components: parameters: max_tokens: 65536 thinking_enable: true + compressor: + backend: ${LLM_BACKEND:-openai} + model: ${LLM_MODEL_NAME:-qwen3.6-flash} + stream: false + context_size: 200000 + max_retries: 5 + retry_delay: 5.0 + credential: + api_key: ${LLM_API_KEY:-} + base_url: ${LLM_BASE_URL:-} + parameters: + max_tokens: 65536 + thinking_enable: false agent_wrapper: default: diff --git a/reme/config/default.yaml b/reme/config/default.yaml index 98650c55..30b29f1a 100644 --- a/reme/config/default.yaml +++ b/reme/config/default.yaml @@ -211,6 +211,36 @@ jobs: steps: - backend: version_step + app_config: + backend: base + description: "return the effective application config with secrets redacted" + parameters: + type: object + properties: { } + steps: + - backend: app_config_step + + chat: + backend: stream + description: "Stream a read-only agent conversation over the ReMe workspace." + parameters: + type: object + properties: + query: + type: string + description: "user message" + session_id: + type: string + description: "agent session to resume" + system_prompt: + type: string + description: "optional system prompt override" + required: + - query + steps: + - backend: chat_step + agent_wrapper: default + health_check: backend: base description: "return a concise health-check snapshot of reme components" @@ -258,19 +288,29 @@ jobs: traverse: backend: base - description: "Walk the wikilink graph from a path." + description: >- + Return a bounded wikilink graph with frontend-ready nodes and directed edges. parameters: type: object properties: path: - type: string - description: "path" + oneOf: + - type: string + - type: array + items: + type: string + minItems: 1 + description: "workspace-relative path or paths used as graph seeds" depth: type: integer - description: "hop limit" + minimum: 0 + description: "maximum number of wikilink hops" default: 1 direction: type: string + description: >- + traversal direction; returned edges always preserve the original + wikilink source-to-target direction enum: - forward - backward @@ -281,6 +321,15 @@ jobs: steps: - backend: traverse_step + graph_snapshot: + backend: base + description: "Return the category-rooted digest wikilink graph with daily-note leaves." + parameters: + type: object + properties: { } + steps: + - backend: graph_snapshot_step + reindex: backend: base description: "wipe the file store and rebuild it from the existing files" @@ -523,6 +572,21 @@ jobs: with_neighbors: false max_neighbors_per_direction: 10 + load: + backend: base + description: "Load a complete text file for the local editor without agent-output truncation." + parameters: + type: object + properties: + path: + type: string + description: "workspace-relative path" + required: + - path + steps: + - backend: load_step + max_bytes: 5242880 + read_image: backend: base description: "Read an image file as base64 (workspace-relative path)." @@ -569,6 +633,27 @@ jobs: steps: - backend: write_step + save: + backend: base + description: "Save text verbatim, optionally rejecting external changes since the file was opened." + parameters: + type: object + properties: + path: + type: string + description: "workspace-relative path" + content: + type: string + description: "complete file content, including frontmatter" + expected_mtime: + type: string + description: "optional ISO mtime returned by stat" + required: + - path + - content + steps: + - backend: save_step + daily_write: backend: base description: "Write a daily markdown note with conversation source frontmatter." diff --git a/reme/config/lme.yaml b/reme/config/lme.yaml index a83d19e4..1fb4f751 100644 --- a/reme/config/lme.yaml +++ b/reme/config/lme.yaml @@ -163,6 +163,28 @@ jobs: steps: - backend: lme_auto_memory_step + # ── Text compression (direct LLM call, no agent) ── + compressor: + backend: base + description: "Compress text via a direct LLM call, optionally guided by queries as relevance filter" + parameters: + type: object + properties: + text: + type: string + description: "the text to compress" + queries: + type: array + description: "optional list of queries; content potentially relevant to any query is kept, content certainly irrelevant to all queries may be dropped" + items: + type: string + default: [] + required: + - text + steps: + - backend: compressor_step + as_llm: compressor + # ── Reindex (full rebuild) ── reindex: backend: base @@ -216,7 +238,7 @@ jobs: - backend: search_v2_step vector_weight: 0.7 candidate_multiplier: 5.0 - expand_links: true + expand_links: false max_links_per_direction: 10 add_draft: @@ -508,6 +530,19 @@ components: parameters: max_tokens: 65536 thinking_enable: true + compressor: + backend: ${LLM_BACKEND:-openai} + model: ${LLM_MODEL_NAME:-qwen3.6-flash} + stream: false + context_size: 200000 + max_retries: 5 + retry_delay: 5.0 + credential: + api_key: ${LLM_API_KEY:-} + base_url: ${LLM_BASE_URL:-} + parameters: + max_tokens: 65536 + thinking_enable: false agent_wrapper: default: diff --git a/reme/schema/__init__.py b/reme/schema/__init__.py index fc919f66..19c8efa8 100644 --- a/reme/schema/__init__.py +++ b/reme/schema/__init__.py @@ -35,12 +35,14 @@ from .dream import ( from .emb_node import EmbNode from .file_chunk import FileChunk from .file_front_matter import FileFrontMatter +from .graph_snapshot import GraphSnapshot, GraphSnapshotEdge, GraphSnapshotNode from .file_link import FileLink from .file_node import FileNode from .request import Request from .response import Response from .stream_chunk import StreamChunk from .token_usage import TokenUsage +from .traverse_graph import TraverseGraph, TraverseGraphEdge, TraverseGraphNode __all__ = [ "ApplicationConfig", @@ -74,6 +76,9 @@ __all__ = [ "FileFrontMatter", "FileLink", "FileNode", + "GraphSnapshot", + "GraphSnapshotEdge", + "GraphSnapshotNode", "IntegrateOutcome", "JobConfig", "PaperInfo", @@ -86,4 +91,7 @@ __all__ = [ "StreamChunk", "TokenUsage", "TopicSelectionOutput", + "TraverseGraph", + "TraverseGraphEdge", + "TraverseGraphNode", ] diff --git a/reme/schema/graph_snapshot.py b/reme/schema/graph_snapshot.py new file mode 100644 index 00000000..44c57f26 --- /dev/null +++ b/reme/schema/graph_snapshot.py @@ -0,0 +1,32 @@ +"""Public response schemas for a complete indexed wikilink graph snapshot.""" + +from typing import Literal + +from pydantic import BaseModel, Field + + +class GraphSnapshotNode(BaseModel): + """One virtual category or indexed Markdown file in a graph snapshot.""" + + id: str = Field(description="Stable node identifier; documents use their workspace-relative path") + path: str = Field(description="Workspace-relative file or virtual category path") + name: str = Field(default="", description="Document name from frontmatter") + description: str = Field(default="", description="Document description from frontmatter") + indexed: bool = Field(description="Whether the target has an indexed FileNode") + virtual: bool = Field(default=False, description="Whether this is a generated category node") + + +class GraphSnapshotEdge(BaseModel): + """One directed category or wikilink edge in a graph snapshot.""" + + source: str = Field(description="Source node identifier") + target: str = Field(description="Target node identifier") + target_anchor: str | None = Field(default=None, description="Optional heading, block, or line anchor") + + +class GraphSnapshot(BaseModel): + """A category-rooted snapshot of digest wikilinks and their daily-note leaves.""" + + version: Literal[1] = 1 + nodes: list[GraphSnapshotNode] + edges: list[GraphSnapshotEdge] diff --git a/reme/schema/traverse_graph.py b/reme/schema/traverse_graph.py new file mode 100644 index 00000000..fe6d5b7d --- /dev/null +++ b/reme/schema/traverse_graph.py @@ -0,0 +1,36 @@ +"""Public response schemas for wikilink graph traversal.""" + +from typing import Literal + +from pydantic import BaseModel, Field + + +class TraverseGraphNode(BaseModel): + """One indexed file or unresolved wikilink target in a traversal result.""" + + id: str = Field(description="Stable node identifier; equal to the workspace-relative path") + path: str = Field(description="Workspace-relative file path") + name: str = Field(default="", description="Document name from frontmatter") + description: str = Field(default="", description="Document description from frontmatter") + depth: int = Field(ge=0, description="Shortest hop distance from any seed") + indexed: bool = Field(description="Whether the target has an indexed FileNode") + + +class TraverseGraphEdge(BaseModel): + """One directed wikilink, preserving its original source and target.""" + + source: str = Field(description="Workspace-relative source file path") + target: str = Field(description="Workspace-relative target file path") + target_anchor: str | None = Field(default=None, description="Optional heading, block, or line anchor") + depth: int = Field(ge=1, description="Traversal depth at which this edge was first reached") + + +class TraverseGraph(BaseModel): + """A bounded wikilink graph rooted at one or more workspace paths.""" + + version: Literal[1] = 1 + seeds: list[str] = Field(description="Normalized workspace-relative traversal roots") + depth: int = Field(ge=0, description="Requested hop limit") + direction: Literal["forward", "backward", "both"] + nodes: list[TraverseGraphNode] + edges: list[TraverseGraphEdge] diff --git a/reme/steps/benchmark/base/agentic_answer.py b/reme/steps/benchmark/base/agentic_answer.py index 27b9f26c..53c854f4 100644 --- a/reme/steps/benchmark/base/agentic_answer.py +++ b/reme/steps/benchmark/base/agentic_answer.py @@ -13,6 +13,10 @@ class BaseAgenticAnswerStep(BaseStep): Subclasses only need to set: TOOL_CONTEXT_PREFIX (str): prefix used to build the unique tool_context_id. + JOB_TOOLS (list[str]): tools exposed to the ReAct agent; override to customize. + INJECTED_JOB_KWARGS (dict): server-owned kwargs injected into every job + tool call via ``injected_job_kwargs``; override the attribute or the + ``_injected_job_kwargs`` hook to customize. And apply their own ``@R.register(...)`` decorator and docstring. @@ -29,6 +33,29 @@ class BaseAgenticAnswerStep(BaseStep): # backend's iteration-counting semantics; other backends ignore it. MAX_ITERATION = 10 TOOL_CONTEXT_PREFIX: str = "content_agentic_answer" + JOB_TOOLS: list[str] = ["search", "add_draft", "read_all_draft", "read"] + INJECTED_JOB_KWARGS: dict = {"read_step_format_session": True} + + def _injected_job_kwargs(self, query: str) -> dict: # pylint: disable=unused-argument + """Server-owned kwargs injected into every job tool call. + + Overridable hook: subclasses can extend the static + ``INJECTED_JOB_KWARGS`` with per-request values derived from ``query``. + + When the runtime context carries a truthy ``compress_session`` flag, + session-transcript compression is enabled in ``search_v2_step`` by + injecting a ``_search._compress.session`` marker plus the current + ``query`` as the query-aware relevance filter. Default (falsy) leaves + session chunks uncompressed. + """ + injected = dict(self.INJECTED_JOB_KWARGS) + if self.context is not None and self.context.get("compress_session"): + injected["_search"] = { + "_compress": {"session": "true"}, + "queries": [query], + "type": "query-aware", + } + return injected async def execute(self): assert self.context is not None @@ -54,10 +81,12 @@ class BaseAgenticAnswerStep(BaseStep): tool_context_id = f"{self.TOOL_CONTEXT_PREFIX}_{os.getpid()}_local" wrapper_kwargs = { "system_prompt": sys_prompt, - "job_tools": ["search", "add_draft", "read_all_draft"], + "job_tools": list(self.JOB_TOOLS), "react_config": {"max_iters": self.MAX_ITERATION}, "tool_context_id": tool_context_id, } + if injected_job_kwargs := self._injected_job_kwargs(f"{query}(query time: {query_time})"): + wrapper_kwargs["injected_job_kwargs"] = injected_job_kwargs if self.context.stream: text = await self._stream_reply(query, **wrapper_kwargs) diff --git a/reme/steps/benchmark/beam/agentic_answer.yaml b/reme/steps/benchmark/beam/agentic_answer.yaml index ab4f00aa..153d211b 100644 --- a/reme/steps/benchmark/beam/agentic_answer.yaml +++ b/reme/steps/benchmark/beam/agentic_answer.yaml @@ -4,12 +4,16 @@ system_prompt: | ## Search Strategy - You can call 'search' tool to search multiple times (at least once) with different queries to gather comprehensive information. + - Do not stop after the first relevant one unless you are confident that the information is sufficient. + - You can use 'read' tool to read specific parts of a specific file. + - After finding a preliminary answer, continue searching from multiple angles to check for potentially conflicting, supplementary, or updated information. ## Draft Tool - Use 'add_draft' to save key findings during search, and 'read_all_draft' to review all saved notes before answering. ## Answer Rules - Answer based ONLY on retrieved context. - Output ONLY the direct factual answer — no reasoning, no search process, no elaboration. - - If information is not founded or not sufficient after multiple searches, reply: 'Information not found.' + - Do NOT invent or assume information that isn't in the memories. Do NOT extrapolate or fabricate connections from related but non-direct evidence. + - If information is not found or not sufficient after multiple searches, or if the maximum iteration count is reached without finding answer, reply: 'Information not found.' temporal_hint: | Current time context: {query_time} diff --git a/reme/steps/benchmark/beam/auto_memory.py b/reme/steps/benchmark/beam/auto_memory.py index b0013dc5..cccb0c28 100644 --- a/reme/steps/benchmark/beam/auto_memory.py +++ b/reme/steps/benchmark/beam/auto_memory.py @@ -5,8 +5,61 @@ from datetime import datetime, timedelta from agentscope.message import Msg from ...evolve.auto_memory import AutoMemoryStep, _normalize_msg_timestamp +from ...file_io import validate_session_id from ....components import R +# Runtime-context key carrying the 0-based line offset of the current segment +# inside the full session file (segmented ingestion of long sessions). +_LINE_OFFSET_KEY = "beam_line_offset" + + +def _msg_word_count(msg: Msg) -> int: + """Whitespace-separated word count of a message's text content.""" + return len((msg.get_text_content() or "").split()) + + +def split_turn_segments(messages: list[Msg], max_words: int) -> list[tuple[int, list[Msg]]]: + """Split *messages* into segments of complete turns bounded by *max_words*. + + A turn starts at every ``user`` message and spans all following non-user + messages (assistant replies). Segments only break at turn boundaries, so a + ``user + assistant`` exchange is never split across two segments. A single + turn larger than *max_words* still becomes its own (oversized) segment. + + Returns ``[(offset, msgs), ...]`` where ``offset`` is the 0-based index of + the segment's first message in *messages* — i.e. its line number in the + session file minus 1. ``max_words <= 0`` disables splitting. + """ + if not messages: + return [] + if max_words <= 0: + return [(0, list(messages))] + + # Group into turns: each user message opens a new turn. + turns: list[list[Msg]] = [] + for msg in messages: + if msg.role == "user" or not turns: + turns.append([msg]) + else: + turns[-1].append(msg) + + segments: list[tuple[int, list[Msg]]] = [] + current: list[Msg] = [] + current_words = 0 + offset = 0 + for turn in turns: + turn_words = sum(_msg_word_count(m) for m in turn) + if current and current_words + turn_words > max_words: + segments.append((offset, current)) + offset += len(current) + current = [] + current_words = 0 + current.extend(turn) + current_words += turn_words + if current: + segments.append((offset, current)) + return segments + def _parse_iso_seconds(value: str) -> datetime | None: """Parse an ISO-8601 timestamp that is precise to at least seconds. @@ -121,8 +174,76 @@ class BeamAutoMemoryStep(AutoMemoryStep): Date pinning for ``daily_write`` is handled by the base class through the agent wrapper's server-owned ``injected_job_kwargs``. + + Long sessions are ingested incrementally: the message list is split into + segments of complete turns (``max_segment_words`` step config, default + 10000 words per segment) and each segment runs one agent pass — the first + pass creates the daily note, later passes merge into it. Line numbers + shown to the agent always refer to the ORIGINAL session file, not the + segment. """ + async def execute(self): + assert self.context is not None + raw_messages = self.context.get("messages") or [] + session_id: str = self.context.get("session_id", "") + max_words = int(self.kwargs.get("max_segment_words", 10000) or 0) + + messages = self._build_messages(raw_messages) + segments = split_turn_segments(messages, max_words) + + if len(segments) <= 1: + self.context[_LINE_OFFSET_KEY] = 0 + await super().execute() + return + + # Persist the FULL session file up-front so every segment's [Ln] labels + # (and the note's source markers) match the final on-disk layout. The + # per-segment saves inside the base execute become no-op appends. + if session_id and validate_session_id(session_id) is None: + await self._save_session_messages(session_id, messages) + + base_hint = self.context.get("memory_hint", "") or "" + answers: list[str] = [] + failed_segments = 0 + self.logger.info( + f"[{self.name}] segmented ingestion session_id={session_id!r} " + f"messages={len(messages)} segments={len(segments)} max_segment_words={max_words}", + ) + for part, (offset, segment) in enumerate(segments, start=1): + part_hint = ( + f"This is part {part}/{len(segments)} of one long session " + f"(lines {offset + 1}-{offset + len(segment)} of the session file). " + "Earlier parts were already recorded; extract ONLY from the turns shown below." + ) + self.context["messages"] = segment + self.context["memory_hint"] = f"{base_hint}\n{part_hint}".strip() + self.context[_LINE_OFFSET_KEY] = offset + # One bad segment (e.g. transiently corrupted note frontmatter) must + # not lose the remaining segments of the session. + try: + await super().execute() + ok = bool(self.context.response.success) + summary = self.context.response.answer or "" + except Exception as exc: # pylint: disable=broad-exception-caught + ok = False + summary = f"Error: {exc}" + self.logger.warning( + f"[{self.name}] segment {part}/{len(segments)} failed session_id={session_id!r}: {exc}", + ) + if not ok: + failed_segments += 1 + answers.append(f"[part {part}/{len(segments)}] {summary}") + + # Restore caller-visible context and aggregate the response. + self.context["messages"] = raw_messages + self.context["memory_hint"] = base_hint + self.context.response.success = failed_segments < len(segments) + self.context.response.answer = "\n".join(answers) + self.context.response.metadata.update( + {"n_messages": len(messages), "segments": len(segments), "failed_segments": failed_segments}, + ) + def _build_messages(self, raw_messages: list) -> list[Msg]: # Interpolate timestamps: if any message carries created_at, fill in # the rest via linear interpolation so the whole session has coherent @@ -131,3 +252,40 @@ class BeamAutoMemoryStep(AutoMemoryStep): [item if not isinstance(item, dict) else dict(item) for item in raw_messages], ) return [self._to_msg(item) for item in interpolated] + + def _format_history(self, messages: list[Msg]) -> str: + # Annotate every turn with its physical line number in the session + # file so the agent can cite information sources as + # [[#L-L]]. A BEAM batch is already + # in chronological order, matching the one-message-per-line layout + # written by AutoMemoryStep._save_session_messages, so number each turn + # by its position plus the segment's offset into the full session file + # (no re-sorting). When the session is ingested in segments, the offset + # keeps [Ln] labels aligned with the ORIGINAL file, not the segment. + session_id = self.context.get("session_id", "") if self.context is not None else "" + line_offset = int(self.context.get(_LINE_OFFSET_KEY, 0) or 0) if self.context is not None else 0 + source_path = self._session_source_path(session_id) + + turns: list[str] = [] + for line, msg in enumerate(messages, start=line_offset + 1): + text = (msg.get_text_content() or "").strip() + if not text: + continue + speaker = msg.name or msg.role or "?" + turns.append(f"[L{line} | {speaker} @ {msg.created_at}]\n{text}") + + if not turns: + return "(empty)" + + first_line = line_offset + 1 + last_line = line_offset + len(messages) + preamble = ( + f"Source file: {source_path}\n" + f"(Excerpt covering lines {first_line}-{last_line} of the source file. Each turn below is " + "prefixed with [Ln] = its line number in the source file; copy these numbers verbatim " + "into source markers.)" + ) + return preamble + "\n\n" + "\n\n".join(turns) + + def _reply_extra_kwargs(self, day: str) -> dict: + return {"tool_defaults": {"daily_write": {"date": day}}} diff --git a/reme/steps/benchmark/beam/auto_memory.yaml b/reme/steps/benchmark/beam/auto_memory.yaml index a9c59c27..7b8421ac 100644 --- a/reme/steps/benchmark/beam/auto_memory.yaml +++ b/reme/steps/benchmark/beam/auto_memory.yaml @@ -7,6 +7,13 @@ system_prompt: | 2. **Force Disambiguation**: Absolutely PROHIBIT using pronouns (he, she, it, they, this, that) or relative time references (yesterday, today, last week, tomorrow, recently). Always use full names, explicit dates, and specific references. 3. **Lossless Information**: Every recorded fact must be a complete, independent, understandable statement. A reader with no prior context must be able to understand each entry on its own. 4. **Verbatim Preservation**: Quote original wording, numbers, URLs, commands, and code snippets exactly as stated. + 5. **Source Attribution**: Mark the source of factual content with a wikilink-style marker built from the `Source file` path (copied verbatim) and the `[Ln]` line numbers of the turns the fact was derived from. The EXACT format is `[[#L-L(,L-L)*]]`, e.g. `[[session/dialog/beam_1M_1_batch1.jsonl#L1-L2]]`. + - **No `Source` label**: write the marker DIRECTLY at the end of the entry (or heading). Do NOT prefix it with `Source`, `Source:`, `source`, `来源`, or any other word — output only the bare `[[...]]` marker. + - **Double brackets + `#` + `L`-prefixed hyphen ranges**: EVERY marker MUST be wrapped in DOUBLE square brackets `[[...]]`, begin with the full session file path, then a hash `#`, then one or more line ranges. Each range is `L-L` — BOTH endpoints carry the `L` prefix and are joined by a hyphen `-` (NEVER a colon). A single line is written as a one-line range, e.g. `L41-L41`. This applies to EVERY comma-separated range too: `L69-L72,L77` is INVALID — write `L69-L72,L77-L77`. Multiple ranges of the same file go into ONE marker, separated by commas with NO spaces. ✗ WRONG: `[session/dialog/....jsonl:29-34]` (old single-bracket colon format), `[[...jsonl#29-34]]` (numbers missing `L`), `[[...jsonl#L29-34]]` (end missing `L`), `[[...jsonl#L29:L34]]` (colon instead of hyphen), `[[L29-L34]]` (path dropped), `[[...jsonl#L41]]` (bare single line — write `L41-L41`), `[[...jsonl#L69-L72,L77]]` (second range is a bare line — write `L77-L77`), `Source: [[...jsonl#L29-L34]]` (label added). ✓ RIGHT: `[[session/dialog/beam_1M_2_batch1.jsonl#L29-L34,L53-L58]]`. + - **Line numbers come from the `[Ln]` labels**: the `[Ln]` label prefixing each input turn (e.g. `[L29 | user @ ...]`) is that message's line number in the ORIGINAL source file — copy those numbers verbatim. The excerpt you see may START at a line far greater than 1; NEVER renumber from 1 and NEVER cite line numbers outside the excerpt's stated range. Newlines inside a message's own content do NOT affect its line number. + - **Merge same-file ranges into ONE marker**: instead of back-to-back markers like `[[...jsonl#L1-L2]][[...jsonl#L5-L6]]`, write a single marker with comma-separated ranges `[[...jsonl#L1-L2,L5-L6]]`. + - **Hoist shared markers, never repeat**: when several consecutive bullets share the SAME marker, do NOT append it to every bullet. Attach ONE marker to the group instead — either append it to the section heading (where it then covers every bullet beneath that heading until a different marker appears), or merge those bullets into a single entry carrying one trailing marker. Repeating an identical marker across sibling bullets is PROHIBITED — it wastes tokens. + - **Cite enough self-contained context**: the cited line range must be self-contained. Do NOT cite only the single line that states the key detail; widen the range to include the surrounding turns (e.g. the question or setup turn that gives the fact its meaning) so that a reader who opens only that line range understands the fact correctly and is not misled by missing context. ## What to Extract @@ -22,11 +29,11 @@ system_prompt: | ## Body Format - Use structured sections with headings. Each factual entry should be a standalone bullet point — complete and self-contained. Group entries by topic. Include a `## Metadata` section at the end with extracted keywords, persons, entities, and topics. + Use structured sections with headings. Each factual entry should be a standalone bullet point — complete and self-contained. Group entries by topic. Attach source markers coarsely (see the Source Attribution principle): prefer ONE marker per section heading or per grouped entry, write only the bare `[[...]]` marker with no `Source` label, and never repeat an identical marker across consecutive bullets. Do NOT add a `## Metadata` section. ## Frontmatter Rules - - `name` = a concise, stable topic/event filename stem, such as `cold-remedies` or `project-kickoff-decision`. Do not include today's date or the daily directory date; the outer daily path already records the date. For existing notes, update it when a better filename is clearly warranted. + - `name` = a concise, stable topic/event filename stem in kebab-case, such as `cold-remedies` or `project-kickoff-decision`. No date, no slash/backslash, none of `< > : " | ? *`. Do not include today's date or the daily directory date; the outer daily path already records the date. For existing notes, update it when a better filename is clearly warranted. - `description` = a thorough, lossless summary; vague descriptions like "notes" / "misc" are unacceptable. The description alone should convey all key information. - **Never set `status`** — it is a field reserved for downstream processing. @@ -37,6 +44,7 @@ system_prompt: | - **NEVER** write tool invocations as text strings such as `daily_write name=...`. Text output does NOT execute tools — no memory will be saved. - **ALWAYS** use the tool call mechanism provided by the runtime. The system handles parameter passing automatically. - If you write a tool name and its parameters as text, NO tool will be executed and NO memory will be saved. + system_prompt_zh: | 你是自动记忆提取系统。你的任务是从对话中提取所有有价值的信息，并记录到结构化的日记记忆中。确保**完整覆盖**——每一条值得保留的信息都必须被捕获。 @@ -46,6 +54,13 @@ system_prompt_zh: | 2. **强制消歧**：绝对禁止使用代词（他、她、它、他们、这个、那个）或相对时间（昨天、今天、上周、明天、最近）。必须使用全名、明确日期和具体引用。 3. **无损信息**：每条记录的事实必须是完整的、独立的、可理解的陈述。没有上下文的读者必须能单独理解每条记录。 4. **原文保留**：逐字引用原始措辞、数字、URL、命令和代码片段。 + 5. **来源标注**：为事实内容标注来源，使用 wikilink 风格的标记，由对话中显示的 `Source file` 路径（逐字复制）与该事实所依据对话轮次的 `[Ln]` 行号拼成。格式**必须**为 `[[#L-L(,L-L)*]]`，例如 `[[session/dialog/beam_1M_1_batch1.jsonl#L1-L2]]`。 + - **不要 `Source` 标签**：把标记**直接**写在条目（或标题）末尾。不要在前面加 `Source`、`Source:`、`source`、`来源` 或任何其他词——只输出裸的 `[[...]]` 标记。 + - **双层方括号 + `#` + 带 `L` 前缀的连字符区间**：每个标记都**必须**用双层方括号 `[[...]]` 包裹，以完整的 session 文件路径开头，然后接 `#`，再接一个或多个行号区间。每个区间写作 `L-L`——两端都带 `L` 前缀，用连字符 `-` 连接（**绝不**用冒号）。单行引用写成单行区间，如 `L41-L41`；逗号分隔的每一个区间也同样适用：`L69-L72,L77` 是**无效的**，应写成 `L69-L72,L77-L77`。同一文件的多个区间放入**同一个**标记，用逗号分隔且不加空格。✗ 错误：`[session/dialog/....jsonl:29-34]`（旧的单括号冒号格式）、`[[...jsonl#29-34]]`（数字缺 `L`）、`[[...jsonl#L29-34]]`（末端缺 `L`）、`[[...jsonl#L29:L34]]`（用了冒号）、`[[L29-L34]]`（丢了路径）、`[[...jsonl#L41]]`（裸单行，应写 `L41-L41`）、`[[...jsonl#L69-L72,L77]]`（第二个区间是裸单行，应写 `L77-L77`）、`Source: [[...jsonl#L29-L34]]`（加了标签）。✓ 正确：`[[session/dialog/beam_1M_2_batch1.jsonl#L29-L34,L53-L58]]`。 + - **行号来自 `[Ln]` 标签**：输入中每条对话轮次前的 `[Ln]` 标签（如 `[L29 | user @ ...]`）就是该消息在**原始**源文件中的行号——请逐字复制这些数字。你看到的片段可能从远大于 1 的行号开始；**绝不**要从 1 重新编号，也**绝不**要引用片段声明范围之外的行号。消息正文内部的换行不影响其行号。 + - **同一文件多区间合并进一个标记**：不要写成相邻的多个标记 `[[...jsonl#L1-L2]][[...jsonl#L5-L6]]`，而要写成用逗号分隔区间的单个标记 `[[...jsonl#L1-L2,L5-L6]]`。 + - **上提共享标记，绝不重复**：当连续多个条目共享**相同**标记时，不要给每个条目都追加。而是给整组只标注一个标记——要么把标记追加到分节标题上（此时它覆盖该标题下、直到出现不同标记之前的所有条目），要么把这些条目合并为一条并只在末尾带一个标记。在同级条目上重复相同标记是**被禁止的**——会浪费 token。 + - **引用充分且自洽的上下文**：所标注的行号区间必须自洽完整。不要只标注陈述关键信息的那一句，要把区间扩大到包含周边轮次（例如赋予该事实意义的提问或铺垫轮次），使得仅打开该行号区间的读者也能正确理解这条事实，不会因缺失上下文而断章取义。 ## 提取什么 @@ -61,11 +76,11 @@ system_prompt_zh: | ## 正文格式 - 使用带标题的结构化分节。每条事实记录应是独立的列表项——完整且自包含。按主题分组。在末尾添加 `## Metadata` 部分，列出提取的关键词、人物、实体和主题。 + 使用带标题的结构化分节。每条事实记录应是独立的列表项——完整且自包含。按主题分组。粗粒度地标注来源（见来源标注原则）：优先每个分节标题或每组条目只用一个标记，只写裸的 `[[...]]` 标记且不带 `Source` 标签，并且绝不在连续条目上重复相同标记。不要添加 `## Metadata` 部分。 ## Frontmatter 规则 - - `name` = 简洁、稳定的主题/事件文件名 stem，例如 `cold-remedies` 或 `project-kickoff-decision`。不要包含今天日期或日记目录日期；外层日记路径已经记录日期。对已有笔记，如果明显有更好的文件名，就更新它。 + - `name` = 简洁、稳定的主题/事件文件名 stem，采用 kebab-case，例如 `cold-remedies` 或 `project-kickoff-decision`。不含日期、不含 slash/反斜杠、不含 `< > : " | ? *`。不要包含今天日期或日记目录日期；外层日记路径已经记录日期。对已有笔记，如果明显有更好的文件名，就更新它。 - `description` = 详尽的无损总结；模糊的描述如 "notes" / "misc" 不可接受。仅凭 description 就应能传达全部核心信息。 - **永远不要设置 `status`**——它是下游处理保留的字段。 @@ -81,6 +96,7 @@ user_message_create: | Today: {today} Extra hint: {note} Session ID: {session_id} + Session file: {session_file} ← copy THIS exact path into every source marker (e.g. `[[{session_file}#L1-L2,L5-L6]]`) # Recent Conversation @@ -115,15 +131,8 @@ user_message_create: | ⚠️ CRITICAL: You MUST invoke `daily_write` as a **tool call** (function call). Do NOT write `daily_write name=...` as plain text — that will NOT execute the tool and NO memory will be saved. - Body requirements: - - Each factual entry must be a complete, independent, self-contained statement (lossless restatement). - - PROHIBIT pronouns (he/she/it/they/this/that) — use full names and explicit references. - - PROHIBIT relative time (yesterday/today/last week) — use absolute dates (YYYY-MM-DD). - - Group entries by topic with headings. - - End with a `## Metadata` section containing: keywords, persons, entities, topic. - - Filename and description rules: - - `name`: concise, stable topic/event filename stem in kebab-case. No date, no slash/backslash, no `< > : " | ? *`. + Body requirements — follow the system prompt's Extraction Principles, Body Format, and Frontmatter Rules (lossless entries; no pronouns; absolute dates; grouped by topic; no `## Metadata`). Two rules that MUST NOT be violated: + - Source markers: put a bare `[[session/dialog/....jsonl#L1-L2,L5-L6]]` at the END of an entry or heading — double brackets, full path + `#`, `L`-prefixed hyphen ranges, no `Source` label, coarse (hoist a shared marker, never repeat it across bullets), and cite a self-contained line range copied verbatim from the `[Ln]` labels. - `description`: a thorough, lossless summary — specific enough that the description alone conveys all key information. ## Step 3 — Summary @@ -137,6 +146,7 @@ user_message_create_zh: | 今天：{today} 额外提示：{note} Session ID：{session_id} + Session 文件：{session_file} ← 请把这个完整路径原样复制进每个来源标记（例如 `[[{session_file}#L1-L2,L5-L6]]`） # 最近的对话 @@ -171,15 +181,8 @@ user_message_create_zh: | ⚠️ 关键：你**必须**以**工具调用**（函数调用）方式调用 `daily_write`。不要将 `daily_write name=...` 作为纯文本输出——那样**不会**执行工具，也**不会**保存任何记忆。 - 正文要求： - - 每条事实记录必须是完整的、独立的、自包含的陈述（无损复述）。 - - 禁止使用代词（他/她/它/他们/这个/那个）——必须使用全名和明确引用。 - - 禁止使用相对时间（昨天/今天/上周）——必须使用绝对日期（YYYY-MM-DD）。 - - 按主题分组并使用标题。 - - 末尾添加 `## Metadata` 部分，包含：keywords、persons、entities、topic。 - - 文件名和描述规则： - - `name`：简洁、稳定的主题/事件文件名 stem，采用 kebab-case。不含日期、不含 slash/反斜杠、不含 `< > : " | ? *`。 + 正文要求——遵循 system prompt 的提取原则、正文格式与 Frontmatter 规则（无损；不用代词；绝对日期；按主题分组；无 `## Metadata`）。以下两条**绝不能违反**： + - 来源标记：在条目或标题**末尾**放一个裸的 `[[session/dialog/....jsonl#L1-L2,L5-L6]]`——双层方括号、完整路径 + `#`、带 `L` 前缀的连字符区间、无 `Source` 标签、粒度粗（上提共享标记、绝不在连续条目上重复），且行号逐字复制自 `[Ln]` 标签、引用自洽完整的行号区间。 - `description`：详尽的无损总结——具体到仅凭 description 就能传达全部核心信息。 ## 步骤 3 — 总结 @@ -194,6 +197,7 @@ user_message_update: | Today: {today} Extra hint: {note} Target path: {note_path} + Session file: {session_file} ← copy THIS exact path into every source marker (e.g. `[[{session_file}#L1-L2,L5-L6]]`) # Recent Conversation @@ -219,19 +223,16 @@ user_message_update: | The note already has content. Extract new information from the conversation and merge into it. - Extraction rules (apply to ALL new entries): - - Each new fact must be a complete, independent, self-contained statement (lossless restatement). - - PROHIBIT pronouns (he/she/it/they/this/that) — use full names and explicit references. - - PROHIBIT relative time (yesterday/today/last week) — use absolute dates (YYYY-MM-DD). + Extraction rules (apply to ALL new entries) — follow the system prompt's Extraction Principles and Body Format (lossless; no pronouns; absolute dates; grouped by topic). Source markers MUST be a bare `[[session/dialog/....jsonl#L1-L2,L5-L6]]` — double brackets, full path + `#`, `L`-prefixed hyphen ranges copied verbatim from the `[Ln]` labels, no `Source` label, coarse (never repeated across bullets), self-contained range. Merge rules: - **Timeline / history entries**: append only, never delete existing entries. - **Current-state entries** (progress, blockers, next steps, open questions): rewrite the entire section to reflect the latest snapshot. - **Everything else**: merge and deduplicate — keep all old facts, add new facts, remove exact duplicates. - - **Metadata section**: merge keywords, persons, entities, topics — union of old and new values. Execution: 1. **Call the `edit` tool** for each section that needs updating, with parameters: `path`="{note_path}", `old`=, `new`=. You may call `edit` multiple times. Each call MUST be a tool call, not text. + - ⚠️ `edit` is for the note BODY only. NEVER touch the frontmatter block (the `---`-delimited YAML at the top of the file) with `edit` — a malformed YAML edit corrupts the whole note. All frontmatter changes MUST go through the `frontmatter_update` tool. 2. After body changes, **call the `frontmatter_update` tool** to refresh frontmatter, with parameters: `path`="{note_path}", `metadata`={{"name": "", "description": ""}}. - Keep the existing `name` only when it is already the best concise topic/event filename stem. The system will rename the file after your final response. - Do not add today's date or the daily directory date to `name`. @@ -241,16 +242,7 @@ user_message_update: | The file exists but its body is empty. Extract and write the full content by **calling the `write` tool** with parameters: `path`="{note_path}", `name`=, `description`=, `content`=. - Body requirements: - - Each factual entry must be a complete, independent, self-contained statement. - - PROHIBIT pronouns — use full names and explicit references. - - PROHIBIT relative time — use absolute dates (YYYY-MM-DD). - - Group entries by topic with headings. - - End with a `## Metadata` section containing: keywords, persons, entities, topic. - - Filename and description rules: - - `name`: concise, stable topic/event filename stem in kebab-case. No date. - - `description`: a thorough, lossless summary — the description alone should convey all key information. + Body requirements — follow the system prompt's Extraction Principles, Body Format, and Frontmatter Rules (lossless; no pronouns; absolute dates; grouped by topic; no `## Metadata`). Source markers MUST be a bare `[[session/dialog/....jsonl#L1-L2,L5-L6]]` — double brackets, full path + `#`, `L`-prefixed hyphen ranges copied verbatim from the `[Ln]` labels, no `Source` label, coarse (never repeated across bullets), self-contained range. `description` must be a thorough, lossless summary that alone conveys all key information. ## Step 4 — Summary @@ -265,6 +257,7 @@ user_message_update_zh: | 今天：{today} 额外提示：{note} 目标路径：{note_path} + Session 文件：{session_file} ← 请把这个完整路径原样复制进每个来源标记（例如 `[[{session_file}#L1-L2,L5-L6]]`） # 最近的对话 @@ -290,19 +283,16 @@ user_message_update_zh: | 笔记已有内容。从对话中提取新信息并合并进去。 - 提取规则（适用于所有新条目）： - - 每条新事实必须是完整的、独立的、自包含的陈述（无损复述）。 - - 禁止使用代词（他/她/它/他们/这个/那个）——必须使用全名和明确引用。 - - 禁止使用相对时间（昨天/今天/上周）——必须使用绝对日期（YYYY-MM-DD）。 + 提取规则（适用于所有新条目）——遵循 system prompt 的提取原则与正文格式（无损；不用代词；绝对日期；按主题分组）。来源标记必须是裸的 `[[session/dialog/....jsonl#L1-L2,L5-L6]]`——双层方括号、完整路径 + `#`、带 `L` 前缀的连字符区间（行号逐字复制自 `[Ln]` 标签）、无 `Source` 标签、粒度粗（不在连续条目上重复）、自洽完整的行号区间。 合并规则： - **时间线 / 历史条目**：仅追加，永远不删除已有条目。 - **当下状态类条目**（进度、卡点、下一步、未决问题）：整段重写，反映最新快照。 - **其余内容**：合并去重——保留全部旧事实，添加新事实，去除完全重复项。 - - **Metadata 部分**：合并 keywords、persons、entities、topics——取旧值和新值的并集。 执行： 1. **调用 `edit` 工具**更新需要修改的部分，参数为：`path`="{note_path}"、`old`=<原文片段>、`new`=<替换片段>。可以多次调用 `edit`。每次调用必须是工具调用，不是文本。 + - ⚠️ `edit` 只能用于笔记**正文**。**绝不**要用 `edit` 碰 frontmatter 块（文件顶部由 `---` 包围的 YAML）——格式错误的 YAML 会损坏整个笔记。所有 frontmatter 修改都必须通过 `frontmatter_update` 工具。 2. 正文变更后，**调用 `frontmatter_update` 工具**刷新 frontmatter，参数为：`path`="{note_path}"、`metadata`={{"name": "<更新后的文件名 stem>", "description": "<更新后的总结>"}}。 - 只有当前 `name` 已经是最合适的简洁主题/事件文件名 stem 时才保留。系统会在你最终回复后负责重命名文件。 - 不要在 `name` 中加入今天日期或日记目录日期。 @@ -312,16 +302,7 @@ user_message_update_zh: | 文件存在但正文为空。提取并**调用 `write` 工具**一次性写入完整内容，参数为：`path`="{note_path}"、`name`=、`description`=、`content`=<正文>。 - 正文要求： - - 每条事实记录必须是完整的、独立的、自包含的陈述。 - - 禁止使用代词——必须使用全名和明确引用。 - - 禁止使用相对时间——必须使用绝对日期（YYYY-MM-DD）。 - - 按主题分组并使用标题。 - - 末尾添加 `## Metadata` 部分，包含：keywords、persons、entities、topic。 - - 文件名和描述规则： - - `name`：简洁、稳定的主题/事件文件名 stem，采用 kebab-case。不含日期。 - - `description`：详尽的无损总结——仅凭 description 就应能传达全部核心信息。 + 正文要求——遵循 system prompt 的提取原则、正文格式与 Frontmatter 规则（无损；不用代词；绝对日期；按主题分组；无 `## Metadata`）。来源标记必须是裸的 `[[session/dialog/....jsonl#L1-L2,L5-L6]]`——双层方括号、完整路径 + `#`、带 `L` 前缀的连字符区间（行号逐字复制自 `[Ln]` 标签）、无 `Source` 标签、粒度粗（不在连续条目上重复）、自洽完整的行号区间。`description` 必须是仅凭自身即可传达全部核心信息的详尽无损总结。 ## 步骤 4 — 总结 diff --git a/reme/steps/benchmark/beam/llm_judge.py b/reme/steps/benchmark/beam/llm_judge.py index bf51f345..879c82e3 100644 --- a/reme/steps/benchmark/beam/llm_judge.py +++ b/reme/steps/benchmark/beam/llm_judge.py @@ -14,7 +14,7 @@ A ``semantic`` alignment path is also available which uses ReMe's configured ``as_embedding`` model (replacing BEAM's sentence_transformers). This replicates the evaluation logic from -``benchmark/datasets/BEAM/src/evaluation/compute_metrics.py``. +``benchmark/beam/dataset/BEAM/src/evaluation/compute_metrics.py``. """ import json diff --git a/reme/steps/benchmark/lme/agentic_answer.py b/reme/steps/benchmark/lme/agentic_answer.py index add3a676..70f8926f 100644 --- a/reme/steps/benchmark/lme/agentic_answer.py +++ b/reme/steps/benchmark/lme/agentic_answer.py @@ -11,6 +11,10 @@ class LmeAgenticAnswerStep(BaseAgenticAnswerStep): The agent uses the ``agent_wrapper`` component in ReAct mode, calling the ``search`` job tool to retrieve relevant memory chunks before generating a final answer. + + Session-transcript compression in ``search_v2_step`` is controlled by the + ``compress_session`` flag in the runtime context (set by the benchmark + runner from ``evaluation.compress_session``); it is off by default. """ TOOL_CONTEXT_PREFIX = "lme_agentic_answer" diff --git a/reme/steps/benchmark/lme/agentic_answer.yaml b/reme/steps/benchmark/lme/agentic_answer.yaml index f856f0fc..9d125086 100644 --- a/reme/steps/benchmark/lme/agentic_answer.yaml +++ b/reme/steps/benchmark/lme/agentic_answer.yaml @@ -1,18 +1,28 @@ system_prompt: | - You are a memory retrieval assistant. You MUST use the search tool to find information before answering. - - Your total time of tool calls should be at most 9 times + You are a memory retrieval assistant. A memory system manages the user's conversation history. Given a query, you MUST call the `search` tool at least once to retrieve relevant content from the memory system before answering. + + ## Memory System + The memory system stores the user's history in three forms: + - **Summary notes**: condensed knowledge distilled from past sessions. + - **Compressed session chunks**: fragments of session transcripts compressed to drop low-information phrasing; the dialogue structure is generally kept, but some detail may be shortened or lost. + - **Original sessions**: complete verbatim transcripts of past conversations, stored as memory files. + + The `search` tool retrieves summary notes and compressed session chunks; it does NOT return original sessions directly. Every result is a chunk — a fragment of a memory file identified by its path — and may lack surrounding context. Compressed session chunks are marked with a leading `compressed session chunk:` notice. When a chunk looks highly relevant to the query but its content is vague, truncated, or missing detail due to compression, use the `read` tool with the chunk's path to read the original session before drawing conclusions. + + ## Workflow + 1. Use `search` to locate possibly relevant content. It is a hybrid retrieval tool combining BM25 keyword matching and vector semantic search, ensuring diverse recall. Call it multiple times with different query phrasings to gather comprehensive information; do not stop after the first relevant result unless you are confident the information is sufficient. + 2. When a retrieved chunk is strongly related to the query but ambiguous because of compression, call `read` on its path to recover the full original session, and ground your answer on that. + 3. Use `add_draft` to save key findings during the search, and `read_all_draft` to review all saved notes before answering. + 4. After finding a preliminary answer, continue searching from multiple angles to check for potentially conflicting, supplementary, or updated information. The goal is to improve the accuracy and completeness of the final answer. + 5. Stop searching and generate the final answer once you are confident that further searches will not yield additional useful information. + - Your total tool calls should be at most 9 times. - ## Search Strategy - - You can call 'search' tool to search multiple times (at least once) with different queries to gather comprehensive information. - - Do not stop after the first relevant one unless you are confident that the information is sufficient. - ## Draft Tool - - Use 'add_draft' to save key findings during search, and 'read_all_draft' to review all saved notes before answering. ## Answer Rules - - Answer based ONLY on retrieved context. - - If the memories contain contradictory information, integrate the context, query, and most importantly the timestamp of each memory to comprehensively determine the final answer. + - Answer based ONLY on retrieved or read context. + - If memories contain contradictory information, weigh the context, query, and most importantly the timestamp of each memory to determine the final answer. - Output ONLY the direct factual answer — no reasoning, no search process, no elaboration. - - Do NOT invent or assume information that isn't in the memories. - - If information is not founded or not sufficient after multiple searches, reply: 'Information not found.' + - Do NOT invent or assume information that isn't in the memories. Do NOT extrapolate or fabricate connections from related but non-direct evidence. + - If information is not found or not sufficient after multiple searches, or if the maximum iteration count is reached without finding answer, reply: 'Information not found.' temporal_hint: | Current time context: {query_time} diff --git a/reme/steps/benchmark/lme/auto_memory.py b/reme/steps/benchmark/lme/auto_memory.py index a27cb45c..bf3a678b 100644 --- a/reme/steps/benchmark/lme/auto_memory.py +++ b/reme/steps/benchmark/lme/auto_memory.py @@ -131,3 +131,31 @@ class LmeAutoMemoryStep(AutoMemoryStep): [item if not isinstance(item, dict) else dict(item) for item in raw_messages], ) return [self._to_msg(item) for item in interpolated] + + def _format_history(self, messages: list[Msg]) -> str: + # Annotate every turn with its physical line number in the session + # file so the agent can cite information sources as + # [[#L-L]]. A LongMemEval session is + # already in chronological order, matching the one-message-per-line + # layout written by AutoMemoryStep._save_session_messages, so number + # each turn by its position in the list (no re-sorting). + session_id = self.context.get("session_id", "") if self.context is not None else "" + source_path = self._session_source_path(session_id) + + turns: list[str] = [] + for line, msg in enumerate(messages, start=1): + text = (msg.get_text_content() or "").strip() + if not text: + continue + speaker = msg.name or msg.role or "?" + turns.append(f"[L{line} | {speaker} @ {msg.created_at}]\n{text}") + + if not turns: + return "(empty)" + + preamble = ( + f"Source file: {source_path}\n" + f"(Each turn below is prefixed with [Ln] = its line number in the source file; " + "copy these numbers verbatim into source markers.)" + ) + return preamble + "\n\n" + "\n\n".join(turns) diff --git a/reme/steps/benchmark/lme/auto_memory.yaml b/reme/steps/benchmark/lme/auto_memory.yaml index a9c59c27..78e88bb9 100644 --- a/reme/steps/benchmark/lme/auto_memory.yaml +++ b/reme/steps/benchmark/lme/auto_memory.yaml @@ -7,6 +7,13 @@ system_prompt: | 2. **Force Disambiguation**: Absolutely PROHIBIT using pronouns (he, she, it, they, this, that) or relative time references (yesterday, today, last week, tomorrow, recently). Always use full names, explicit dates, and specific references. 3. **Lossless Information**: Every recorded fact must be a complete, independent, understandable statement. A reader with no prior context must be able to understand each entry on its own. 4. **Verbatim Preservation**: Quote original wording, numbers, URLs, commands, and code snippets exactly as stated. + 5. **Source Attribution**: Mark the source of factual content with a wikilink-style marker built from the `Source file` path (copied verbatim) and the `[Ln]` line numbers of the turns the fact was derived from. The EXACT format is `[[#L-L(,L-L)*]]`, e.g. `[[session/dialog/s1.jsonl#L1-L2]]`. + - **No `Source` label**: write the marker DIRECTLY at the end of the entry (or heading). Do NOT prefix it with `Source`, `Source:`, `source`, `来源`, or any other word — output only the bare `[[...]]` marker. + - **Double brackets + `#` + `L`-prefixed hyphen ranges**: EVERY marker MUST be wrapped in DOUBLE square brackets `[[...]]`, begin with the full session file path, then a hash `#`, then one or more line ranges. Each range is `L-L` — BOTH endpoints carry the `L` prefix and are joined by a hyphen `-` (NEVER a colon). A single line is written as a one-line range, e.g. `L41-L41`. This applies to EVERY comma-separated range too: `L69-L72,L77` is INVALID — write `L69-L72,L77-L77`. Multiple ranges of the same file go into ONE marker, separated by commas with NO spaces. ✗ WRONG: `[session/dialog/....jsonl:29-34]` (old single-bracket colon format), `[[...jsonl#29-34]]` (numbers missing `L`), `[[...jsonl#L29-34]]` (end missing `L`), `[[...jsonl#L29:L34]]` (colon instead of hyphen), `[[L29-L34]]` (path dropped), `[[...jsonl#L41]]` (bare single line — write `L41-L41`), `[[...jsonl#L69-L72,L77]]` (second range is a bare line — write `L77-L77`), `Source: [[...jsonl#L29-L34]]` (label added). ✓ RIGHT: `[[session/dialog/s1.jsonl#L29-L34,L53-L58]]`. + - **Line numbers come from the `[Ln]` labels**: the `[Ln]` label prefixing each input turn (e.g. `[L29 | user @ ...]`) is that message's line number in the ORIGINAL source file — copy those numbers verbatim. Newlines inside a message's own content do NOT affect its line number. + - **Merge same-file ranges into ONE marker**: instead of back-to-back markers like `[[...jsonl#L1-L2]][[...jsonl#L5-L6]]`, write a single marker with comma-separated ranges `[[...jsonl#L1-L2,L5-L6]]`. + - **Hoist shared markers, never repeat**: when several consecutive bullets share the SAME marker, do NOT append it to every bullet. Attach ONE marker to the group instead — either append it to the section heading (where it then covers every bullet beneath that heading until a different marker appears), or merge those bullets into a single entry carrying one trailing marker. Repeating an identical marker across sibling bullets is PROHIBITED — it wastes tokens. + - **Cite enough self-contained context**: the cited line range must be self-contained. Do NOT cite only the single line that states the key detail; widen the range to include the surrounding turns (e.g. the question or setup turn that gives the fact its meaning) so that a reader who opens only that line range understands the fact correctly and is not misled by missing context. ## What to Extract @@ -22,7 +29,7 @@ system_prompt: | ## Body Format - Use structured sections with headings. Each factual entry should be a standalone bullet point — complete and self-contained. Group entries by topic. Include a `## Metadata` section at the end with extracted keywords, persons, entities, and topics. + Use structured sections with headings. Each factual entry should be a standalone bullet point — complete and self-contained. Group entries by topic. Attach source markers coarsely (see the Source Attribution principle): prefer ONE marker per section heading or per grouped entry, write only the bare `[[...]]` marker with no `Source` label, and never repeat an identical marker across consecutive bullets. Do NOT add a `## Metadata` section. ## Frontmatter Rules @@ -46,6 +53,13 @@ system_prompt_zh: | 2. **强制消歧**：绝对禁止使用代词（他、她、它、他们、这个、那个）或相对时间（昨天、今天、上周、明天、最近）。必须使用全名、明确日期和具体引用。 3. **无损信息**：每条记录的事实必须是完整的、独立的、可理解的陈述。没有上下文的读者必须能单独理解每条记录。 4. **原文保留**：逐字引用原始措辞、数字、URL、命令和代码片段。 + 5. **来源标注**：为事实内容标注来源，使用 wikilink 风格的标记，由对话中显示的 `Source file` 路径（逐字复制）与该事实所依据对话轮次的 `[Ln]` 行号拼成。格式**必须**为 `[[#L-L(,L-L)*]]`，例如 `[[session/dialog/s1.jsonl#L1-L2]]`。 + - **不要 `Source` 标签**：把标记**直接**写在条目（或标题）末尾。不要在前面加 `Source`、`Source:`、`source`、`来源` 或任何其他词——只输出裸的 `[[...]]` 标记。 + - **双层方括号 + `#` + 带 `L` 前缀的连字符区间**：每个标记都**必须**用双层方括号 `[[...]]` 包裹，以完整的 session 文件路径开头，然后接 `#`，再接一个或多个行号区间。每个区间写作 `L-L`——两端都带 `L` 前缀，用连字符 `-` 连接（**绝不**用冒号）。单行引用写成单行区间，如 `L41-L41`；逗号分隔的每一个区间也同样适用：`L69-L72,L77` 是**无效的**，应写成 `L69-L72,L77-L77`。同一文件的多个区间放入**同一个**标记，用逗号分隔且不加空格。✗ 错误：`[session/dialog/....jsonl:29-34]`（旧的单括号冒号格式）、`[[...jsonl#29-34]]`（数字缺 `L`）、`[[...jsonl#L29-34]]`（末端缺 `L`）、`[[...jsonl#L29:L34]]`（用了冒号）、`[[L29-L34]]`（丢了路径）、`[[...jsonl#L41]]`（裸单行，应写 `L41-L41`）、`[[...jsonl#L69-L72,L77]]`（第二个区间是裸单行，应写 `L77-L77`）、`Source: [[...jsonl#L29-L34]]`（加了标签）。✓ 正确：`[[session/dialog/s1.jsonl#L29-L34,L53-L58]]`。 + - **行号来自 `[Ln]` 标签**：输入中每条对话轮次前的 `[Ln]` 标签（如 `[L29 | user @ ...]`）就是该消息在**原始**源文件中的行号——请逐字复制这些数字。消息正文内部的换行不影响其行号。 + - **同一文件多区间合并进一个标记**：不要写成相邻的多个标记 `[[...jsonl#L1-L2]][[...jsonl#L5-L6]]`，而要写成用逗号分隔区间的单个标记 `[[...jsonl#L1-L2,L5-L6]]`。 + - **上提共享标记，绝不重复**：当连续多个条目共享**相同**标记时，不要给每个条目都追加。而是给整组只标注一个标记——要么把标记追加到分节标题上（此时它覆盖该标题下、直到出现不同标记之前的所有条目），要么把这些条目合并为一条并只在末尾带一个标记。在同级条目上重复相同标记是**被禁止的**——会浪费 token。 + - **引用充分且自洽的上下文**：所标注的行号区间必须自洽完整。不要只标注陈述关键信息的那一句，要把区间扩大到包含周边轮次（例如赋予该事实意义的提问或铺垫轮次），使得仅打开该行号区间的读者也能正确理解这条事实，不会因缺失上下文而断章取义。 ## 提取什么 @@ -61,7 +75,7 @@ system_prompt_zh: | ## 正文格式 - 使用带标题的结构化分节。每条事实记录应是独立的列表项——完整且自包含。按主题分组。在末尾添加 `## Metadata` 部分，列出提取的关键词、人物、实体和主题。 + 使用带标题的结构化分节。每条事实记录应是独立的列表项——完整且自包含。按主题分组。粗粒度地标注来源（见来源标注原则）：优先每个分节标题或每组条目只用一个标记，只写裸的 `[[...]]` 标记且不带 `Source` 标签，并且绝不在连续条目上重复相同标记。不要添加 `## Metadata` 部分。 ## Frontmatter 规则 @@ -81,6 +95,7 @@ user_message_create: | Today: {today} Extra hint: {note} Session ID: {session_id} + Session file: {session_file} ← copy THIS exact path into every source marker (e.g. `[[{session_file}#L1-L2,L5-L6]]`) # Recent Conversation @@ -115,15 +130,8 @@ user_message_create: | ⚠️ CRITICAL: You MUST invoke `daily_write` as a **tool call** (function call). Do NOT write `daily_write name=...` as plain text — that will NOT execute the tool and NO memory will be saved. - Body requirements: - - Each factual entry must be a complete, independent, self-contained statement (lossless restatement). - - PROHIBIT pronouns (he/she/it/they/this/that) — use full names and explicit references. - - PROHIBIT relative time (yesterday/today/last week) — use absolute dates (YYYY-MM-DD). - - Group entries by topic with headings. - - End with a `## Metadata` section containing: keywords, persons, entities, topic. - - Filename and description rules: - - `name`: concise, stable topic/event filename stem in kebab-case. No date, no slash/backslash, no `< > : " | ? *`. + Body requirements — follow the system prompt's Extraction Principles, Body Format, and Frontmatter Rules (lossless entries; no pronouns; absolute dates; grouped by topic; no `## Metadata`). Two rules that MUST NOT be violated: + - Source markers: put a bare `[[session/dialog/....jsonl#L1-L2,L5-L6]]` at the END of an entry or heading — double brackets, full path + `#`, `L`-prefixed hyphen ranges, no `Source` label, coarse (hoist a shared marker, never repeat it across bullets), and cite a self-contained line range copied verbatim from the `[Ln]` labels. - `description`: a thorough, lossless summary — specific enough that the description alone conveys all key information. ## Step 3 — Summary @@ -137,6 +145,7 @@ user_message_create_zh: | 今天：{today} 额外提示：{note} Session ID：{session_id} + Session 文件：{session_file} ← 请把这个完整路径原样复制进每个来源标记（例如 `[[{session_file}#L1-L2,L5-L6]]`） # 最近的对话 @@ -171,15 +180,8 @@ user_message_create_zh: | ⚠️ 关键：你**必须**以**工具调用**（函数调用）方式调用 `daily_write`。不要将 `daily_write name=...` 作为纯文本输出——那样**不会**执行工具，也**不会**保存任何记忆。 - 正文要求： - - 每条事实记录必须是完整的、独立的、自包含的陈述（无损复述）。 - - 禁止使用代词（他/她/它/他们/这个/那个）——必须使用全名和明确引用。 - - 禁止使用相对时间（昨天/今天/上周）——必须使用绝对日期（YYYY-MM-DD）。 - - 按主题分组并使用标题。 - - 末尾添加 `## Metadata` 部分，包含：keywords、persons、entities、topic。 - - 文件名和描述规则： - - `name`：简洁、稳定的主题/事件文件名 stem，采用 kebab-case。不含日期、不含 slash/反斜杠、不含 `< > : " | ? *`。 + 正文要求——遵循 system prompt 的提取原则、正文格式与 Frontmatter 规则（无损；不用代词；绝对日期；按主题分组；无 `## Metadata`）。以下两条**绝不能违反**： + - 来源标记：在条目或标题**末尾**放一个裸的 `[[session/dialog/....jsonl#L1-L2,L5-L6]]`——双层方括号、完整路径 + `#`、带 `L` 前缀的连字符区间、无 `Source` 标签、粒度粗（上提共享标记、绝不在连续条目上重复），且行号逐字复制自 `[Ln]` 标签、引用自洽完整的行号区间。 - `description`：详尽的无损总结——具体到仅凭 description 就能传达全部核心信息。 ## 步骤 3 — 总结 @@ -194,6 +196,7 @@ user_message_update: | Today: {today} Extra hint: {note} Target path: {note_path} + Session file: {session_file} ← copy THIS exact path into every source marker (e.g. `[[{session_file}#L1-L2,L5-L6]]`) # Recent Conversation @@ -219,19 +222,16 @@ user_message_update: | The note already has content. Extract new information from the conversation and merge into it. - Extraction rules (apply to ALL new entries): - - Each new fact must be a complete, independent, self-contained statement (lossless restatement). - - PROHIBIT pronouns (he/she/it/they/this/that) — use full names and explicit references. - - PROHIBIT relative time (yesterday/today/last week) — use absolute dates (YYYY-MM-DD). + Extraction rules (apply to ALL new entries) — follow the system prompt's Extraction Principles and Body Format (lossless; no pronouns; absolute dates; grouped by topic). Source markers MUST be a bare `[[session/dialog/....jsonl#L1-L2,L5-L6]]` — double brackets, full path + `#`, `L`-prefixed hyphen ranges copied verbatim from the `[Ln]` labels, no `Source` label, coarse (never repeated across bullets), self-contained range. Merge rules: - **Timeline / history entries**: append only, never delete existing entries. - **Current-state entries** (progress, blockers, next steps, open questions): rewrite the entire section to reflect the latest snapshot. - **Everything else**: merge and deduplicate — keep all old facts, add new facts, remove exact duplicates. - - **Metadata section**: merge keywords, persons, entities, topics — union of old and new values. Execution: 1. **Call the `edit` tool** for each section that needs updating, with parameters: `path`="{note_path}", `old`=, `new`=. You may call `edit` multiple times. Each call MUST be a tool call, not text. + - ⚠️ `edit` is for the note BODY only. NEVER touch the frontmatter block (the `---`-delimited YAML at the top of the file) with `edit` — a malformed YAML edit corrupts the whole note. All frontmatter changes MUST go through the `frontmatter_update` tool. 2. After body changes, **call the `frontmatter_update` tool** to refresh frontmatter, with parameters: `path`="{note_path}", `metadata`={{"name": "", "description": ""}}. - Keep the existing `name` only when it is already the best concise topic/event filename stem. The system will rename the file after your final response. - Do not add today's date or the daily directory date to `name`. @@ -241,16 +241,7 @@ user_message_update: | The file exists but its body is empty. Extract and write the full content by **calling the `write` tool** with parameters: `path`="{note_path}", `name`=, `description`=, `content`=. - Body requirements: - - Each factual entry must be a complete, independent, self-contained statement. - - PROHIBIT pronouns — use full names and explicit references. - - PROHIBIT relative time — use absolute dates (YYYY-MM-DD). - - Group entries by topic with headings. - - End with a `## Metadata` section containing: keywords, persons, entities, topic. - - Filename and description rules: - - `name`: concise, stable topic/event filename stem in kebab-case. No date. - - `description`: a thorough, lossless summary — the description alone should convey all key information. + Body requirements — follow the system prompt's Extraction Principles, Body Format, and Frontmatter Rules (lossless; no pronouns; absolute dates; grouped by topic; no `## Metadata`). Source markers MUST be a bare `[[session/dialog/....jsonl#L1-L2,L5-L6]]` — double brackets, full path + `#`, `L`-prefixed hyphen ranges copied verbatim from the `[Ln]` labels, no `Source` label, coarse (never repeated across bullets), self-contained range. `description` must be a thorough, lossless summary that alone conveys all key information. ## Step 4 — Summary @@ -265,6 +256,7 @@ user_message_update_zh: | 今天：{today} 额外提示：{note} 目标路径：{note_path} + Session 文件：{session_file} ← 请把这个完整路径原样复制进每个来源标记（例如 `[[{session_file}#L1-L2,L5-L6]]`） # 最近的对话 @@ -290,19 +282,16 @@ user_message_update_zh: | 笔记已有内容。从对话中提取新信息并合并进去。 - 提取规则（适用于所有新条目）： - - 每条新事实必须是完整的、独立的、自包含的陈述（无损复述）。 - - 禁止使用代词（他/她/它/他们/这个/那个）——必须使用全名和明确引用。 - - 禁止使用相对时间（昨天/今天/上周）——必须使用绝对日期（YYYY-MM-DD）。 + 提取规则（适用于所有新条目）——遵循 system prompt 的提取原则与正文格式（无损；不用代词；绝对日期；按主题分组）。来源标记必须是裸的 `[[session/dialog/....jsonl#L1-L2,L5-L6]]`——双层方括号、完整路径 + `#`、带 `L` 前缀的连字符区间（行号逐字复制自 `[Ln]` 标签）、无 `Source` 标签、粒度粗（不在连续条目上重复）、自洽完整的行号区间。 合并规则： - **时间线 / 历史条目**：仅追加，永远不删除已有条目。 - **当下状态类条目**（进度、卡点、下一步、未决问题）：整段重写，反映最新快照。 - **其余内容**：合并去重——保留全部旧事实，添加新事实，去除完全重复项。 - - **Metadata 部分**：合并 keywords、persons、entities、topics——取旧值和新值的并集。 执行： 1. **调用 `edit` 工具**更新需要修改的部分，参数为：`path`="{note_path}"、`old`=<原文片段>、`new`=<替换片段>。可以多次调用 `edit`。每次调用必须是工具调用，不是文本。 + - ⚠️ `edit` 只能用于笔记**正文**。**绝不**要用 `edit` 碰 frontmatter 块（文件顶部由 `---` 包围的 YAML）——格式错误的 YAML 会损坏整个笔记。所有 frontmatter 修改都必须通过 `frontmatter_update` 工具。 2. 正文变更后，**调用 `frontmatter_update` 工具**刷新 frontmatter，参数为：`path`="{note_path}"、`metadata`={{"name": "<更新后的文件名 stem>", "description": "<更新后的总结>"}}。 - 只有当前 `name` 已经是最合适的简洁主题/事件文件名 stem 时才保留。系统会在你最终回复后负责重命名文件。 - 不要在 `name` 中加入今天日期或日记目录日期。 @@ -312,16 +301,7 @@ user_message_update_zh: | 文件存在但正文为空。提取并**调用 `write` 工具**一次性写入完整内容，参数为：`path`="{note_path}"、`name`=、`description`=、`content`=<正文>。 - 正文要求： - - 每条事实记录必须是完整的、独立的、自包含的陈述。 - - 禁止使用代词——必须使用全名和明确引用。 - - 禁止使用相对时间——必须使用绝对日期（YYYY-MM-DD）。 - - 按主题分组并使用标题。 - - 末尾添加 `## Metadata` 部分，包含：keywords、persons、entities、topic。 - - 文件名和描述规则： - - `name`：简洁、稳定的主题/事件文件名 stem，采用 kebab-case。不含日期。 - - `description`：详尽的无损总结——仅凭 description 就应能传达全部核心信息。 + 正文要求——遵循 system prompt 的提取原则、正文格式与 Frontmatter 规则（无损；不用代词；绝对日期；按主题分组；无 `## Metadata`）。来源标记必须是裸的 `[[session/dialog/....jsonl#L1-L2,L5-L6]]`——双层方括号、完整路径 + `#`、带 `L` 前缀的连字符区间（行号逐字复制自 `[Ln]` 标签）、无 `Source` 标签、粒度粗（不在连续条目上重复）、自洽完整的行号区间。`description` 必须是仅凭自身即可传达全部核心信息的详尽无损总结。 ## 步骤 4 — 总结 diff --git a/reme/steps/common/__init__.py b/reme/steps/common/__init__.py index 61c55167..50b5def1 100644 --- a/reme/steps/common/__init__.py +++ b/reme/steps/common/__init__.py @@ -1,6 +1,8 @@ """common steps""" from .add import AddStep +from .app_config import AppConfigStep +from .chat import ChatStep from .demo import DemoEchoStep1, DemoEchoStep2 from .health_check import HealthCheckStep from .help import HelpStep @@ -14,6 +16,8 @@ from .version import VersionStep __all__ = [ "AddStep", + "AppConfigStep", + "ChatStep", "DemoEchoStep1", "DemoEchoStep2", "HealthCheckStep", diff --git a/reme/steps/common/app_config.py b/reme/steps/common/app_config.py new file mode 100644 index 00000000..678bbaa5 --- /dev/null +++ b/reme/steps/common/app_config.py @@ -0,0 +1,85 @@ +"""Return the effective application configuration.""" + +from typing import Any +from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit + +from ..base_step import BaseStep +from ...components import R + +_SECRET_KEYS = { + "api_key", + "app_secret", + "authorization", + "cookie", + "password", + "private_key", + "proxy_authorization", + "secret", + "set_cookie", + "token", +} +_SECRET_SUFFIXES = ("_access_key", "_api_key", "_password", "_private_key", "_secret", "_token") + + +def _is_secret_key(key: str) -> bool: + normalized = key.lower().replace("-", "_") + return normalized in _SECRET_KEYS or normalized.endswith(_SECRET_SUFFIXES) + + +def _redact_url(value: str) -> str: + """Redact credentials embedded in an HTTP-style URL.""" + try: + parsed = urlsplit(value) + if not parsed.scheme or not parsed.netloc: + return value + + netloc = parsed.netloc + if parsed.password is not None: + host = parsed.hostname or "" + if ":" in host: + host = f"[{host}]" + if parsed.port is not None: + host = f"{host}:{parsed.port}" + netloc = f"{parsed.username or ''}:***@{host}" + + query = urlencode( + [ + (key, "***" if _is_secret_key(key) else item) + for key, item in parse_qsl(parsed.query, keep_blank_values=True) + ], + ) + return urlunsplit((parsed.scheme, netloc, parsed.path, query, parsed.fragment)) + except ValueError: + return value + + +def _redact(value: Any) -> Any: + if isinstance(value, dict): + redacted = {} + for key, item in value.items(): + key_string = str(key) + if _is_secret_key(key_string) or ( + key_string.lower() == "credential" and not isinstance(item, (dict, list)) + ): + redacted[key] = "***" + else: + redacted[key] = _redact(item) + return redacted + if isinstance(value, list): + return [_redact(item) for item in value] + if isinstance(value, str): + return _redact_url(value) + return value + + +@R.register("app_config_step") +class AppConfigStep(BaseStep): + """Return the effective config without startup environment or secrets.""" + + async def execute(self): + assert self.context is not None + assert self.app_context is not None + + config = self.app_context.app_config.model_dump(mode="json", exclude={"environment"}) + self.context.response.answer = _redact(config) + return self.context.response diff --git a/reme/steps/common/chat.py b/reme/steps/common/chat.py new file mode 100644 index 00000000..56f18de1 --- /dev/null +++ b/reme/steps/common/chat.py @@ -0,0 +1,45 @@ +"""Workspace-aware streaming chat step for the ReMe web interface.""" + +from ..base_step import BaseStep +from ...components import R +from ...enumeration import ChunkEnum + + +@R.register("chat_step") +class ChatStep(BaseStep): + """Stream a read-only agent conversation over the current workspace.""" + + DEFAULT_SYSTEM_PROMPT = """You are ReMe Agent, an assistant for a local-first memory workspace. +Use the available ReMe tools when workspace facts are needed. Cite workspace-relative file paths +when referring to notes. Never invent file contents, and do not claim to have changed files because +this chat intentionally provides read-only tools. Reply in the user's language.""" + READ_ONLY_TOOLS = ["search", "list", "read", "stat", "traverse"] + + async def execute(self): + assert self.context is not None + query = str(self.context.get("query") or "").strip() + if not query: + self.context.response.success = False + self.context.response.answer = "Skipped: empty query" + return self.context.response + if self.agent_wrapper is None: + raise RuntimeError("chat_step requires an agent_wrapper") + + wrapper_kwargs = { + "system_prompt": self.context.get("system_prompt") or self.DEFAULT_SYSTEM_PROMPT, + "job_tools": self.READ_ONLY_TOOLS, + } + if session_id := self.context.get("session_id"): + wrapper_kwargs["resume"] = str(session_id) + + parts: list[str] = [] + async for chunk in self.agent_wrapper.reply_stream(query, **wrapper_kwargs): + await self.context.add_stream_chunk(chunk) + if chunk.chunk_type == ChunkEnum.CONTENT and isinstance(chunk.chunk, str): + parts.append(chunk.chunk) + if chunk.session_id: + self.context.response.metadata["session_id"] = chunk.session_id + + self.context.response.success = True + self.context.response.answer = "".join(parts).strip() + return self.context.response diff --git a/reme/steps/evolve/__init__.py b/reme/steps/evolve/__init__.py index e2ab5fd6..423b7598 100644 --- a/reme/steps/evolve/__init__.py +++ b/reme/steps/evolve/__init__.py @@ -4,6 +4,7 @@ from ._evolve import now from .auto_memory import AutoMemoryStep from .auto_memory_cc import AutoMemoryCCStep from .auto_resource import AutoResourceStep +from .compressor import CompressorStep from .dream import DreamExtractStep, DreamFinishStep, DreamIntegrateStep, DreamTopicsStep, ProactiveStep __all__ = [ @@ -11,6 +12,7 @@ __all__ = [ "AutoMemoryStep", "AutoMemoryCCStep", "AutoResourceStep", + "CompressorStep", "DreamExtractStep", "DreamFinishStep", "DreamIntegrateStep", diff --git a/reme/steps/evolve/auto_memory.py b/reme/steps/evolve/auto_memory.py index b0ceef88..0277fd5f 100644 --- a/reme/steps/evolve/auto_memory.py +++ b/reme/steps/evolve/auto_memory.py @@ -71,8 +71,11 @@ class AutoMemoryStep(BaseStep): def _session_path(self, session_id: str) -> Path: return self.file_store.workspace_path / self._session_dir() / "dialog" / f"{session_id}.jsonl" + def _session_source_path(self, session_id: str) -> str: + return f"{self._session_dir()}/dialog/{session_id}.jsonl" + def _session_link(self, session_id: str) -> str: - return f"[[{self._session_dir()}/dialog/{session_id}.jsonl]]" + return f"[[{self._session_source_path(session_id)}]]" def _daily_note_path(self, day: str, name: str) -> str: return f"{self.config_value('daily_dir')}/{day}/{name}.md" @@ -240,6 +243,14 @@ class AutoMemoryStep(BaseStep): """ return {} + def _format_history(self, messages: list[Msg]) -> str: + """Render the conversation history injected into the prompt. + + Overridable hook: subclasses can annotate messages (e.g. with their + source line numbers in the session file) before rendering. + """ + return format_history(messages) + # pylint: disable=too-many-return-statements async def execute(self): assert self.context is not None @@ -308,7 +319,8 @@ class AutoMemoryStep(BaseStep): note=memory_hint or "(none)", note_path=note_path, session_id=session_id, - history=format_history(messages), + session_file=self._session_source_path(session_id), + history=self._format_history(messages), ) self.logger.info(f"[{self.name}] agent start path={note_path} template={template_key}") diff --git a/reme/steps/evolve/compressor.py b/reme/steps/evolve/compressor.py new file mode 100644 index 00000000..11832014 --- /dev/null +++ b/reme/steps/evolve/compressor.py @@ -0,0 +1,82 @@ +"""compressor — compress text via a direct LLM call (no agent).""" + +from agentscope.message import UserMsg +from agentscope.model import ChatResponse + +from ..base_step import BaseStep +from ...components import R + + +@R.register("compressor_step") +class CompressorStep(BaseStep): + """Compress ``text`` with a direct LLM call (no agent wrapper). + + Inputs (from RuntimeContext): + text (str, required): the text to compress. + queries (list[str], optional, default []): when provided, compression + keeps anything potentially relevant to any of the queries and may + drop content that is certainly irrelevant and certainly unhelpful + for all of them. + + Output (written to context.response.answer): + The compressed text. + """ + + async def execute(self): + assert self.context is not None + text: str = self.context.get("text", "") or "" + raw_queries = self.context.get("queries") or [] + queries: list[str] = [str(q).strip() for q in raw_queries if str(q).strip()] + + if not text.strip(): + self.context.response.success = False + self.context.response.answer = "Skipped: empty text" + return + + if queries: + queries_block = "\n".join(f"- {q}" for q in queries) + user_message = self.prompt_format("compress_query_prompt", queries=queries_block, text=text) + else: + user_message = self.prompt_format("compress_prompt", text=text) + + result = await self.as_llm([UserMsg(name="user", content=user_message)]) + compressed = await self._response_text(result) + self.logger.info( + f"[{self.name}] compressed {len(text)} -> {len(compressed)} chars (queries={len(queries)})", + ) + + self.context.response.success = True + self.context.response.answer = compressed + self.context.response.metadata.update( + { + "queries": queries, + "original_length": len(text), + "compressed_length": len(compressed), + }, + ) + + @staticmethod + async def _response_text(result) -> str: + """Extract text blocks from a streaming or non-streaming ChatResponse. + + The async-iteration check is done on the type: ``ChatResponse`` inherits + agentscope's ``DictMixin`` whose instance ``__getattr__`` raises + ``KeyError`` (not ``AttributeError``), so ``hasattr(result, ...)`` + would crash instead of returning False. + """ + if hasattr(type(result), "__aiter__"): + last: ChatResponse | None = None + async for chunk in result: + last = chunk + result = last + if result is None: + return "" + # Content blocks may be plain dicts or pydantic models (TextBlock/ThinkingBlock). + parts: list[str] = [] + for block in result.content or []: + if isinstance(block, dict): + if block.get("type") == "text": + parts.append(str(block.get("text") or "")) + elif getattr(block, "type", None) == "text": + parts.append(str(getattr(block, "text", "") or "")) + return "".join(parts).strip() diff --git a/reme/steps/evolve/compressor.yaml b/reme/steps/evolve/compressor.yaml new file mode 100644 index 00000000..5df2c38b --- /dev/null +++ b/reme/steps/evolve/compressor.yaml @@ -0,0 +1,91 @@ +compress_prompt: | + You are a text compression engine. The text below is a conversation transcript. Compress it so it becomes shorter while losing zero information. + + ## Compression Rules + + 1. Delete sentences that carry no real information: greetings, pleasantries, small talk, thanks, apologies, courtesy formulas, filler words, and empty transitions (e.g. "Hi there!", "Hope you're doing well", "Thanks so much!", "As we all know"). + 2. You may rewrite, merge, or restructure sentences to make them shorter, but you must NOT lose any information. Every fact, entity, number, date, condition, causal relation, opinion, preference, and nuance present in the original must remain fully recoverable from your output. + 3. Never delete informative content, and never weaken it into a vaguer statement. Compression must come only from removing zero-information phrasing and from tighter wording. + 4. Never add new information, comments, guesses, or interpretations of your own. + 5. Copy critical details verbatim: numbers, dates, times, amounts, identifiers, proper names, quotes, file paths, and code must appear exactly as in the original. Vague or relative time expressions (e.g. "last Wednesday", "two weeks ago") must also be preserved as-is. + 6. Keep the original language of the text, and try to preserve the original format (such as the `[role @ timestamp]` message lines) and the original order of information. + 7. If the entire text carries no real information (pure greetings/small talk), output nothing. + + ## Output Format + + Output ONLY the compressed text. No explanations, no preamble, no quotes, no markdown fences. + + # Text + + {text} +compress_prompt_zh: | + 你是一个文本压缩引擎。下面的文本是一段对话记录。请压缩它，使其变短，但信息量零损失。 + + ## 压缩规则 + + 1. 删除没有实际信息量的语句：问候、寒暄、客套、感谢、道歉、口头禅、空洞的过渡语（例如"你好呀！""希望你一切都好""非常感谢！""众所周知"）。 + 2. 可以改写、合并、调整语句以减小长度，但不得损失任何信息。原文中的每一个事实、实体、数字、日期、条件、因果关系、观点、偏好和细微含义，都必须能从你的输出中完整还原。 + 3. 不得删减信息内容，也不得把信息弱化成更模糊的表述。压缩只能来自删除零信息量的表达和更紧凑的措辞。 + 4. 不得添加任何新信息、评论、猜测或你自己的解读。 + 5. 关键细节必须逐字保留：数字、日期、时间、金额、标识符、专有名词、引语、文件路径和代码必须与原文完全一致。模糊或相对的时间描述（如"上周三""两周前"）也必须原样保留。 + 6. 保持文本的原始语言，尽量保持原有的格式（如 `[role @ 时间戳]` 的消息行）和信息的原有顺序。 + 7. 如果整段文本没有任何实际信息（纯问候/寒暄），则输出空。 + + ## 输出格式 + + 只输出压缩后的文本。不要解释、不要开场白、不要引号、不要 markdown 代码块。 + + # 文本 + + {text} + +compress_query_prompt: | + You are a text compression engine. The text below is a conversation transcript. Compress it so it becomes shorter, guided by the queries as a relevance filter, without losing any potentially useful information. + + ## Compression Rules + + 1. Delete sentences that carry no real information: greetings, pleasantries, small talk, thanks, apologies, courtesy formulas, filler words, and empty transitions (e.g. "Hi there!", "Hope you're doing well", "Thanks so much!", "As we all know"). + 2. You may rewrite, merge, or restructure sentences to make them shorter, but the kept content must NOT lose information. Every fact, entity, number, date, condition, causal relation, opinion, preference, and nuance you keep must remain fully recoverable from your output. + 3. Keep ALL content that is possibly or potentially related to ANY of the queries. Even weak, indirect, background, or uncertain relevance to any single query means it MUST be kept. + 4. You may delete a piece of information ONLY when both conditions hold for EVERY query: it is certainly unrelated to that query, AND it is certainly of no help whatsoever for answering that query. + 5. If you are unsure whether something can be deleted, KEEP it. When in doubt, always err on the side of keeping. + 6. Do NOT answer the queries. Do NOT add new information, comments, guesses, or interpretations of your own. + 7. Copy critical details verbatim: numbers, dates, times, amounts, identifiers, proper names, quotes, file paths, and code must appear exactly as in the original. Vague or relative time expressions (e.g. "last Wednesday", "two weeks ago") must also be preserved as-is. + 8. Keep the original language of the text, and try to preserve the original format (such as the `[role @ timestamp]` message lines) and the original order of information. + + ## Output Format + + Output ONLY the compressed text. No explanations, no preamble, no quotes, no markdown fences. + + # Queries + + {queries} + + # Text + + {text} +compress_query_prompt_zh: | + 你是一个文本压缩引擎。下面的文本是一段对话记录。请以下列 query 列表作为相关性过滤条件，压缩文本使其变短，同时不丢失任何可能有用的信息。 + + ## 压缩规则 + + 1. 删除没有实际信息量的语句：问候、寒暄、客套、感谢、道歉、口头禅、空洞的过渡语（例如"你好呀！""希望你一切都好""非常感谢！""众所周知"）。 + 2. 可以改写、合并、调整语句以减小长度，但保留下来的内容不得损失信息。你保留的每一个事实、实体、数字、日期、条件、因果关系、观点、偏好和细微含义，都必须能从你的输出中完整还原。 + 3. 保留任何可能、潜在与任意一个 query 相关的内容。哪怕只与其中某一个 query 相关性微弱、间接、只是背景信息或不确定，也必须保留。 + 4. 只有对每一个 query 都同时满足以下两个条件时，才可以删除一条信息：它一定与该 query 无关，并且对回答该 query 一定没有任何帮助。 + 5. 对于不确定是否可以删除的信息，一律保留。拿不准时，永远倾向于保留。 + 6. 不要回答这些 query 本身。不得添加任何新信息、评论、猜测或你自己的解读。 + 7. 关键细节必须逐字保留：数字、日期、时间、金额、标识符、专有名词、引语、文件路径和代码必须与原文完全一致。模糊或相对的时间描述（如"上周三""两周前"）也必须原样保留。 + 8. 保持文本的原始语言，尽量保持原有的格式（如 `[role @ 时间戳]` 的消息行）和信息的原有顺序。 + + ## 输出格式 + + 只输出压缩后的文本。不要解释、不要开场白、不要引号、不要 markdown 代码块。 + + # Query 列表 + + {queries} + + # 文本 + + {text} diff --git a/reme/steps/file_io/__init__.py b/reme/steps/file_io/__init__.py index 932aa6c6..7417e9ed 100644 --- a/reme/steps/file_io/__init__.py +++ b/reme/steps/file_io/__init__.py @@ -12,9 +12,11 @@ from .frontmatter_delete import FrontmatterDeleteStep from .frontmatter_read import FrontmatterReadStep from .frontmatter_update import FrontmatterUpdateStep from .list import ListStep +from .load import LoadStep from .move import MoveStep from .read import ReadStep from .read_image import ReadImageStep +from .save import SaveStep from .stat import StatStep from .write import WriteStep @@ -35,9 +37,11 @@ __all__ = [ "FrontmatterReadStep", "FrontmatterUpdateStep", "ListStep", + "LoadStep", "MoveStep", "ReadStep", "ReadImageStep", + "SaveStep", "StatStep", "WriteStep", ] diff --git a/reme/steps/file_io/_file_io.py b/reme/steps/file_io/_file_io.py index 1a979449..82c5264d 100644 --- a/reme/steps/file_io/_file_io.py +++ b/reme/steps/file_io/_file_io.py @@ -8,6 +8,7 @@ import aiofiles import aiofiles.os from ...constants import DEFAULT_MAX_BYTES, MAX_FILE_READ_BYTES, TRUNCATION_NOTICE_MARKER +from ...schema import FileChunk from ...utils import get_logger logger = get_logger(log_to_file=False) @@ -214,3 +215,44 @@ def truncate_text_output( except Exception: logger.warning("truncate_text_output failed, returning original text", exc_info=True) return text + + +def truncate_session_output( + text: str, + *, + start_line: int = 1, + total_lines: int = 0, + max_bytes: int = DEFAULT_MAX_BYTES, + file_path: str | None = None, + encoding: str = "utf-8", +) -> str: + """Render session jsonl lines then truncate by bytes preserving line integrity. + + Each line of *text* is expected to be a serialized ``Msg`` JSON from a raw + session transcript (``*.jsonl``). Lines are first rendered via + :func:`~reme.steps.index._source_format.render_session_chunk_lines` — one + message per line as ``[speaker @ created_at] content`` with internal + newlines flattened — and then truncated with the same byte-budget logic + as :func:`truncate_text_output`. + """ + if not text or max_bytes <= 0: + return text + + # Render each jsonl line into a human-readable [speaker @ time] content line. + # Import locally to avoid a hard dependency from file_io on the index package + # at module load time; the import is safe (no circular dependency). + from ..index._source_format import render_session_chunk_lines + + chunk = FileChunk(text=text) + rendered_lines = render_session_chunk_lines(chunk) + rendered_text = "\n".join(rendered_lines) + + # Reuse the same byte-budget truncation as truncate_text_output. + return truncate_text_output( + rendered_text, + start_line=start_line, + total_lines=total_lines, + max_bytes=max_bytes, + file_path=file_path, + encoding=encoding, + ) diff --git a/reme/steps/file_io/list.py b/reme/steps/file_io/list.py index 721328a1..ebacc967 100644 --- a/reme/steps/file_io/list.py +++ b/reme/steps/file_io/list.py @@ -68,7 +68,7 @@ class ListStep(BaseStep): out: list[str] = [] for entry in files: try: - out.append(str(entry.relative_to(workspace_dir))) + out.append(entry.relative_to(workspace_dir).as_posix()) except ValueError: out.append(str(entry)) return out diff --git a/reme/steps/file_io/load.py b/reme/steps/file_io/load.py new file mode 100644 index 00000000..94bc23fc --- /dev/null +++ b/reme/steps/file_io/load.py @@ -0,0 +1,66 @@ +"""Load a complete text file for an interactive editor.""" + +from datetime import datetime + +from ._file_io import read_file_safe +from ._path import _check_path_permission, resolve_path +from ..base_step import BaseStep +from ...components import R + +DEFAULT_EDITOR_MAX_BYTES = 5 * 1024 * 1024 + + +@R.register("load_step") +class LoadStep(BaseStep): + """Return complete text without the truncation used by agent-facing reads.""" + + def _fail(self, message: str, **metadata) -> None: + assert self.context is not None + self.context.response.success = False + self.context.response.answer = f"Error: {message}" + self.context.response.metadata.update(metadata) + + async def execute(self): + assert self.context is not None + raw_path = str(self.context.get("path") or "") + max_bytes = int(self.context.get("max_bytes") or self.kwargs.get("max_bytes") or DEFAULT_EDITOR_MAX_BYTES) + target, error = resolve_path(self.workspace_path, raw_path) + if error or target is None: + self._fail(error or "invalid path", path=raw_path) + return self.context.response + if not _check_path_permission(self.workspace_path, target, self.context.get("_allowed_paths")): + self._fail("no permission to read this file", path=raw_path) + return self.context.response + if not target.is_file(): + self._fail("file does not exist", path=raw_path) + return self.context.response + + stat = target.stat() + if stat.st_size > max_bytes: + self._fail( + f"file is too large for the editor ({stat.st_size} bytes; limit {max_bytes})", + code="file_too_large", + path=raw_path, + size=stat.st_size, + max_bytes=max_bytes, + ) + return self.context.response + try: + content, encoding = await read_file_safe(target, max_bytes=max_bytes) + except Exception as exc: # pylint: disable=broad-except + self._fail(f"load failed: {exc}", path=raw_path) + return self.context.response + + self.context.response.success = True + self.context.response.answer = content + self.context.response.metadata.update( + { + "path": raw_path, + "exists": True, + "type": "file", + "size": stat.st_size, + "mtime": datetime.fromtimestamp(stat.st_mtime).isoformat(), + "encoding": encoding, + }, + ) + return self.context.response diff --git a/reme/steps/file_io/read.py b/reme/steps/file_io/read.py index ce530a87..0df2342a 100644 --- a/reme/steps/file_io/read.py +++ b/reme/steps/file_io/read.py @@ -2,9 +2,10 @@ from pathlib import Path -from ._file_io import read_file_lines_safe, read_file_safe, truncate_text_output +from ._file_io import read_file_lines_safe, read_file_safe, truncate_session_output, truncate_text_output from ._path import _check_path_permission, NON_MD_WARNING, gate_md, resolve_path from ..base_step import BaseStep +from ..index._source_format import is_session_path from ...components import R from ...constants import DEFAULT_MAX_BYTES, MAX_FILE_READ_BYTES from ...utils import expand_links, render_expansion_lines @@ -21,6 +22,14 @@ class ReadStep(BaseStep): neighbors (out/in link targets) with name/description meta, fetched via the file_store. Same rendering as SearchStep. max_neighbors_per_direction (int, default 10): cap per direction. + read_step_format_session (bool, default False): when true and the + resolved path is a raw session transcript (``*.jsonl`` under the + dialog dir), render each jsonl line as + ``[speaker @ created_at] content`` before truncation via + :func:`truncate_session_output`. Non-session files fall back to + the standard :func:`truncate_text_output` path. Injected at + runtime via ``injected_job_kwargs`` (takes precedence) or set in + the YAML ``steps`` config as a fallback. Permission: honors the request-scoped ``_allowed_paths`` constraint injected by the server into the RuntimeContext. @@ -31,6 +40,7 @@ class ReadStep(BaseStep): assert self.context is not None self.context.response.success = False self.context.response.answer = f"Error: {message}" + self.logger.warning(f"[{self.name}] FAILED: {message} | meta={meta}") if meta: self.context.response.metadata.update(meta) @@ -92,6 +102,29 @@ class ReadStep(BaseStep): self._fail(f"read failed: {e}", path=str(target)) return None + def _truncate_with_session_format(self, excerpt: str, target: Path, start_line: int, total: int) -> str: + """Truncate *excerpt* using session-aware rendering when *target* is a session file. + + Falls back to :func:`truncate_text_output` when the path is not under + the workspace or is not a raw session transcript. + """ + try: + rel_path = str(target.relative_to(self.workspace_path)) + except ValueError: + self.logger.info(f"[{self.name}] skip session format: path outside workspace_path path={target}") + return truncate_text_output(excerpt, start_line=start_line, total_lines=total, file_path=str(target)) + + dialog_dir = self.config_value("dialog_dir") + if not is_session_path(rel_path, dialog_dir): + return truncate_text_output(excerpt, start_line=start_line, total_lines=total, file_path=str(target)) + + return truncate_session_output( + excerpt, + start_line=start_line, + total_lines=total, + file_path=str(target), + ) + # pylint: disable=too-many-return-statements async def execute(self): assert self.context is not None @@ -99,6 +132,12 @@ class ReadStep(BaseStep): start_line, end_line = self.context.get("start_line"), self.context.get("end_line") with_neighbors: bool = bool(self.kwargs.get("with_neighbors", False)) max_neighbors_per_direction: int = int(self.kwargs.get("max_neighbors_per_direction", 10)) + # Injected value takes precedence over YAML kwargs; check existence + # (not truthiness) so an explicit False can disable a YAML-true flag. + _format_session = self.context.get("read_step_format_session") + if _format_session is None: + _format_session = self.kwargs.get("read_step_format_session", False) + format_session: bool = bool(_format_session) # Validate inputs and target before touching the filesystem twice. target = self._resolve_target(raw) @@ -151,7 +190,10 @@ class ReadStep(BaseStep): return None e = min(total, requested_end if requested_end is not None else total) - text = truncate_text_output(excerpt, start_line=s, total_lines=total, file_path=str(target)) + if format_session: + text = self._truncate_with_session_format(excerpt, target, s, total) + else: + text = truncate_text_output(excerpt, start_line=s, total_lines=total, file_path=str(target)) self.context.response.success = True self.context.response.answer = text diff --git a/reme/steps/file_io/save.py b/reme/steps/file_io/save.py new file mode 100644 index 00000000..0b77cf7a --- /dev/null +++ b/reme/steps/file_io/save.py @@ -0,0 +1,78 @@ +"""Save a text file verbatim with optional optimistic-concurrency protection.""" + +from datetime import datetime + +from ._file_io import detect_file_encoding, get_path_lock, write_file_safe +from ._path import _check_path_permission, resolve_path +from ..base_step import BaseStep +from ...components import R + + +@R.register("save_step") +class SaveStep(BaseStep): + """Persist editor content without interpreting or rebuilding frontmatter. + + ``expected_mtime`` is the ISO timestamp returned by ``stat``. When supplied, + saving fails if the file changed after it was opened. This keeps the + filesystem authoritative when ReMe and another editor are used together. + """ + + def _fail(self, message: str, **metadata) -> None: + assert self.context is not None + self.context.response.success = False + self.context.response.answer = f"Error: {message}" + self.context.response.metadata.update(metadata) + + async def execute(self): + assert self.context is not None + raw_path = str(self.context.get("path") or "") + content = str(self.context.get("content") or "") + expected_mtime = self.context.get("expected_mtime") + + target, error = resolve_path(self.workspace_path, raw_path) + if error or target is None: + self._fail(error or "invalid path", path=raw_path) + return self.context.response + if not _check_path_permission(self.workspace_path, target, self.context.get("_allowed_paths")): + self._fail("no permission to write this file", path=raw_path) + return self.context.response + if target.exists() and not target.is_file(): + self._fail("target is not a file", path=raw_path) + return self.context.response + + lock = await get_path_lock(target) + async with lock: + current_mtime = datetime.fromtimestamp(target.stat().st_mtime).isoformat() if target.exists() else None + if expected_mtime and current_mtime != str(expected_mtime): + self._fail( + "file changed outside ReMe; reload it before saving", + code="file_conflict", + path=raw_path, + expected_mtime=str(expected_mtime), + current_mtime=current_mtime, + ) + return self.context.response + + encoding = await detect_file_encoding(target) if target.exists() else "utf-8" + try: + await write_file_safe(target, content, encoding=encoding) + except Exception as exc: # pylint: disable=broad-except + self._fail(f"save failed: {exc}", path=raw_path) + return self.context.response + + stat = target.stat() + saved_mtime = datetime.fromtimestamp(stat.st_mtime).isoformat() + + self.context.response.success = True + self.context.response.answer = f"Saved {raw_path} ({stat.st_size} bytes)" + self.context.response.metadata.update( + { + "path": raw_path, + "exists": True, + "type": "file", + "size": stat.st_size, + "mtime": saved_mtime, + }, + ) + self.logger.info(f"[{self.name}] saved path={target} bytes={stat.st_size} encoding={encoding}") + return self.context.response diff --git a/reme/steps/index/__init__.py b/reme/steps/index/__init__.py index 3e8a16bd..51727e27 100644 --- a/reme/steps/index/__init__.py +++ b/reme/steps/index/__init__.py @@ -4,6 +4,7 @@ from .bm25_search import Bm25SearchStep from .clear_paths import ClearPathsStep from .clear_store import ClearStoreStep from .draft import AddDraftStep, ReadAllDraftStep +from .graph_snapshot import GraphSnapshotStep from .log_changes import LogChangesStep from .node_search import NodeSearchStep from .init_changes import InitChangesStep @@ -30,6 +31,7 @@ __all__ = [ "DEFAULT_LOW_POWER_POLL_MS", "DEFAULT_WATCH_DEBOUNCE_MS", "DEFAULT_WATCH_STEP_MS", + "GraphSnapshotStep", "InitChangesStep", "LogChangesStep", "NodeSearchStep", diff --git a/reme/steps/index/_source_format.py b/reme/steps/index/_source_format.py index 0e42cb82..d7a46db9 100644 --- a/reme/steps/index/_source_format.py +++ b/reme/steps/index/_source_format.py @@ -1,16 +1,20 @@ """Shared helpers: render retrieved chunks and assemble search-step answers. Raw session transcripts (``*.jsonl`` under the dialog dir) store one serialized -``Msg`` per line. :func:`render_chunk_body` turns those back into a readable -dialog; all other chunks keep their raw ``text``. Used by +``Msg`` per line. :func:`render_chunk_body` renders those line-aligned — one +message per line with internal newlines flattened; all other chunks keep their +raw ``text``. Used by ``search``/``vector_search``/``bm25_search`` so every step renders session hits identically. -:func:`format_chunks_answer` assembles a full answer string from a list of -chunks, with optional per-chunk score formatting and link expansion. Raw -session chunks from the same file whose line ranges overlap, contain one -another, or are adjacent are merged into their union before rendering (see -:func:`_merge_session_chunk_intervals`) so a passage is never shown twice. +:func:`render_chunk_entries` renders each chunk into a ``{"path", +"start_line", "end_line", "score", "body", "link"}`` entry dict, with optional +per-chunk score formatting and link expansion; :func:`join_chunk_entries` +assembles the source headers and joins those entries into the final answer +string. Chunks are rendered as given; callers that want raw session chunks from +the same file with overlapping, contained, or adjacent line ranges collapsed +into their union should pre-merge them via :func:`merge_session_chunk_intervals` +so a passage is never shown twice. :data:`ALL_RETURNED_MESSAGE` is the English notice shown when tool_context dedup removes every previously-returned result. :data:`NO_RESULTS_MESSAGE` is the English notice shown when the search returned no results at all. @@ -20,7 +24,6 @@ from typing import Callable, Final from agentscope.message import Msg -from ..evolve._evolve import format_history from ...schema import FileChunk from ...utils.link_expansion import render_expansion_lines @@ -35,36 +38,59 @@ ALL_RETURNED_MESSAGE: Final[str] = ( NO_RESULTS_MESSAGE: Final[str] = "No relevant information was found for the given query." -def is_session_chunk(chunk: FileChunk, dialog_dir: str) -> bool: - """True if the chunk comes from a raw session transcript (a jsonl file under the dialog dir).""" - path = (chunk.path or "").strip().strip("/") +def is_session_path(path: str, dialog_dir: str) -> bool: + """True if the path points at a raw session transcript (a jsonl file under the dialog dir).""" + path = (path or "").strip().strip("/") if not path.endswith(".jsonl"): return False dialog_dir = (dialog_dir or "").strip("/") return path == dialog_dir or path.startswith(f"{dialog_dir}/") -def render_chunk_body(chunk: FileChunk, dialog_dir: str) -> str: - """Render a chunk's body, compacting raw session transcripts into a readable form. +def is_session_chunk(chunk: FileChunk, dialog_dir: str) -> bool: + """True if the chunk comes from a raw session transcript (a jsonl file under the dialog dir).""" + return is_session_path(chunk.path, dialog_dir) - Session chunks are jsonl where each line is a serialized ``Msg``. Parse every line - and render via :func:`format_history`; on any parse error (or no usable messages), - fall back to the chunk's raw ``text``. + +def render_chunk_body(chunk: FileChunk, dialog_dir: str) -> str: + """Render a chunk's body; raw session transcripts render one message per line. + + Session chunks are jsonl where each line is a serialized ``Msg``. They + render via :func:`render_session_chunk_lines` — one message per line with + internal newlines flattened — so verbatim and compressed session bodies + share the same single-line-per-message format. All other chunks keep + their raw ``text``. """ if not is_session_chunk(chunk, dialog_dir): return chunk.text - try: - messages: list[Msg] = [] - for line in chunk.text.splitlines(): - line = line.strip() - if not line: - continue - messages.append(Msg.model_validate_json(line)) - if not messages: - return chunk.text - return format_history(messages) - except Exception: - return chunk.text + return "\n".join(render_session_chunk_lines(chunk)) + + +def render_session_chunk_lines(chunk: FileChunk) -> list[str]: + """Render a raw session chunk line-aligned with the original jsonl file. + + Output line ``i`` maps to file line ``chunk.start_line + i`` (1-based). + Each jsonl line (one serialized ``Msg`` per line) becomes a single rendered + line ``[speaker @ created_at] content`` with internal whitespace and + newlines flattened to single spaces. Blank lines stay blank and + unparseable lines pass through stripped, so the file-line mapping is + never broken. + """ + rendered: list[str] = [] + for raw in chunk.text.splitlines(): + line = raw.strip() + if not line: + rendered.append("") + continue + try: + msg = Msg.model_validate_json(line) + except Exception: + rendered.append(line) + continue + speaker = msg.name or msg.role or "?" + content = " ".join((msg.get_text_content() or "").split()) + rendered.append(f"[{speaker} @ {msg.created_at}] {content}".rstrip()) + return rendered def _build_union_chunk(group: list[FileChunk]) -> FileChunk: @@ -97,7 +123,7 @@ def _build_union_chunk(group: list[FileChunk]) -> FileChunk: ) -def _merge_session_chunk_intervals(chunks: list[FileChunk], dialog_dir: str) -> list[FileChunk]: +def merge_session_chunk_intervals(chunks: list[FileChunk], dialog_dir: str) -> list[FileChunk]: """Merge raw session chunks from the same file into their line-range union. Only chunks recognized as raw session transcripts (see @@ -155,39 +181,71 @@ def _finalize_group(group: list[FileChunk]) -> FileChunk: return _build_union_chunk(group) -def format_chunks_answer( +def render_chunk_entries( chunks: list[FileChunk], dialog_dir: str, *, include_source: bool = True, score_fn: Callable[[FileChunk], str] | None = None, link_expansion: dict[str, dict] | None = None, -) -> str: - """Render a list of chunks into a single answer string. +) -> list[dict[str, str]]: + """Render each chunk into an entry dict of source fields, body, and link. - Raw session chunks from the same file that overlap, contain one another, or - are adjacent are merged into their union first (see - :func:`_merge_session_chunk_intervals`). + Chunks are rendered exactly as given, in input order. Callers that want + overlapping/contained/adjacent raw session chunks collapsed into their + union should pre-merge the list via :func:`merge_session_chunk_intervals` + before calling. - When *include_source* is ``True`` (default), each chunk is prefixed with a - source header showing path, line range, and score. When ``False``, only the - rendered bodies are included, separated by blank lines. + When *include_source* is ``True`` (default), each entry carries the source + fields ``path`` / ``start_line`` / ``end_line`` / ``score`` (the formatted + score string) plus ``body`` and ``link`` (the per-path expansion lines, + empty string when there is none). When ``False``, each entry carries only + ``body``. - *score_fn* customizes the score string in the header (default - ``"score={chunk.score:.4f}"``). *link_expansion* appends per-path expansion - lines after each chunk's header block (used by hybrid search). + *score_fn* customizes the formatted ``score`` string (default + ``"score={chunk.score:.4f}"``). *link_expansion* provides the per-path + expansion lines (used by hybrid search). """ - chunks = _merge_session_chunk_intervals(chunks, dialog_dir) if not include_source: - return "\n\n".join(render_chunk_body(c, dialog_dir) for c in chunks) + return [{"body": render_chunk_body(c, dialog_dir)} for c in chunks] fmt = score_fn or (lambda c: f"score={c.score:.4f}") - lines: list[str] = [] + entries: list[dict[str, str]] = [] for c in chunks: - lines.append( - f"========== {c.path}:{c.start_line}-{c.end_line} " - f"[{fmt(c)}] ==========\n{render_chunk_body(c, dialog_dir)}", + entries.append( + { + "path": c.path, + "start_line": str(c.start_line), + "end_line": str(c.end_line), + "score": fmt(c), + "body": render_chunk_body(c, dialog_dir), + "link": "\n".join(render_expansion_lines((link_expansion or {}).get(c.path, {}))), + }, ) - if link_expansion: - lines.extend(render_expansion_lines(link_expansion.get(c.path, {}))) - return "\n".join(lines) + return entries + + +def join_chunk_entries(entries: list[dict[str, str]]) -> str: + """Assemble entries into the final answer string. + + For each entry, a source header line is rebuilt from ``path`` / + ``start_line`` / ``end_line`` / ``score`` when ``path`` is present, then + followed by the non-empty ``body`` and ``link`` parts, joined with ``\\n``. + Entries are separated by a blank line; missing or empty parts are skipped. + """ + parts: list[str] = [] + for entry in entries: + tmp = [] + if path := entry.get("path", "").strip(): + tmp.append( + f"========== {path}:{entry.get('start_line', '')}-{entry.get('end_line', '')} " + f"[{entry.get('score', '')}] ==========", + ) + for key in ("body", "link"): + value = entry.get(key, "").strip() + if not value: + continue + tmp.append(value) + if tmp: + parts.append("\n".join(tmp)) + return "\n\n".join(parts) diff --git a/reme/steps/index/bm25_search.py b/reme/steps/index/bm25_search.py index 4fb84e55..f2fc4998 100644 --- a/reme/steps/index/bm25_search.py +++ b/reme/steps/index/bm25_search.py @@ -3,7 +3,8 @@ from typing import Final from ._dedup import _ToolContextDedupMixin -from ._source_format import ALL_RETURNED_MESSAGE, NO_RESULTS_MESSAGE, format_chunks_answer +from ._source_format import ALL_RETURNED_MESSAGE, NO_RESULTS_MESSAGE, join_chunk_entries +from ._source_format import merge_session_chunk_intervals, render_chunk_entries from ..base_step import BaseStep from ...components import R @@ -48,11 +49,12 @@ class Bm25SearchStep(_ToolContextDedupMixin, BaseStep): results = results[:limit] dialog_dir = self.config_value("dialog_dir") - self.context.response.answer = format_chunks_answer( - results, + entries = render_chunk_entries( + merge_session_chunk_intervals(results, dialog_dir), dialog_dir, include_source=self.include_source, ) + self.context.response.answer = join_chunk_entries(entries) if not results: self.context.response.answer = ALL_RETURNED_MESSAGE if pre_dedup_count > 0 else NO_RESULTS_MESSAGE self.context.response.metadata["results"] = [ diff --git a/reme/steps/index/graph_snapshot.py b/reme/steps/index/graph_snapshot.py new file mode 100644 index 00000000..3132db73 --- /dev/null +++ b/reme/steps/index/graph_snapshot.py @@ -0,0 +1,86 @@ +"""Return the digest graph, rooted by its three memory categories.""" + +from ..base_step import BaseStep +from ...components import R +from ...enumeration import DreamBucketEnum +from ...schema import GraphSnapshot, GraphSnapshotEdge, GraphSnapshotNode + +_CATEGORY_BUCKETS = ( + DreamBucketEnum.WIKI, + DreamBucketEnum.PERSONAL, + DreamBucketEnum.PROCEDURE, +) + + +@R.register("graph_snapshot_step") +class GraphSnapshotStep(BaseStep): + """Build the frontend digest graph through the file-store contract. + + Category nodes connect to every indexed Markdown file in their bucket. + Digest files retain wikilinks to other digest files and to daily notes. Daily + notes are leaves: their own outgoing links are intentionally not returned. + """ + + async def execute(self): + assert self.context is not None + indexed_nodes = await self.file_store.get_nodes() + node_by_path = {node.path: node for node in indexed_nodes if node.path.lower().endswith(".md")} + + digest_dir = str(self.config_value("digest_dir")).strip("/") + daily_dir = str(self.config_value("daily_dir")).strip("/") + category_paths = { + bucket: f"{digest_dir}/{bucket.value}" if digest_dir else bucket.value for bucket in _CATEGORY_BUCKETS + } + daily_prefix = f"{daily_dir}/" if daily_dir else "" + + digest_paths_by_category = { + label: sorted(path for path in node_by_path if path.startswith(f"{category_path}/")) + for label, category_path in category_paths.items() + } + digest_paths = {path for paths in digest_paths_by_category.values() for path in paths} + + edge_keys: set[tuple[str, str, str | None]] = set() + for source in digest_paths: + for link in node_by_path[source].links: + target = link.target_path + if target in digest_paths or (target in node_by_path and target.startswith(daily_prefix)): + edge_keys.add((source, target, link.target_anchor)) + + daily_paths = {target for _source, target, _anchor in edge_keys if target.startswith(daily_prefix)} + + nodes = [ + GraphSnapshotNode( + id=f"virtual:{bucket.value}", + path=category_paths[bucket], + name=bucket.value, + indexed=False, + virtual=True, + ) + for bucket in _CATEGORY_BUCKETS + ] + for bucket in _CATEGORY_BUCKETS: + for path in digest_paths_by_category[bucket]: + edge_keys.add((f"virtual:{bucket.value}", path, None)) + + for path in sorted(digest_paths | daily_paths): + node = node_by_path[path] + nodes.append( + GraphSnapshotNode( + id=path, + path=path, + name=node.front_matter.name, + description=node.front_matter.description, + indexed=True, + ), + ) + + edges = [ + GraphSnapshotEdge(source=source, target=target, target_anchor=anchor) + for source, target, anchor in sorted(edge_keys, key=lambda edge: (edge[0], edge[1], edge[2] or "")) + ] + graph = GraphSnapshot(nodes=nodes, edges=edges) + + self.context.response.success = True + self.context.response.answer = graph.model_dump() + self.logger.info(f"[{self.name}] nodes={len(nodes)} edges={len(edges)}") + return self.context.response diff --git a/reme/steps/index/search_v2.py b/reme/steps/index/search_v2.py index 9f2821b1..bdeb91e0 100644 --- a/reme/steps/index/search_v2.py +++ b/reme/steps/index/search_v2.py @@ -2,7 +2,8 @@ This is the local fork of the upstream search step. It uses :class:`_ToolContextDedupMixin` for subset-aware interval-merging dedup and -:func:`format_chunks_answer` for session-aware chunk formatting with +:func:`render_chunk_entries` / :func:`join_chunk_entries` for session-aware +chunk formatting with :data:`ALL_RETURNED_MESSAGE` / :data:`NO_RESULTS_MESSAGE` notices. """ @@ -12,7 +13,8 @@ import os from typing import Final from ._dedup import _ToolContextDedupMixin -from ._source_format import ALL_RETURNED_MESSAGE, NO_RESULTS_MESSAGE, format_chunks_answer +from ._source_format import ALL_RETURNED_MESSAGE, NO_RESULTS_MESSAGE, is_session_path, join_chunk_entries +from ._source_format import merge_session_chunk_intervals, render_chunk_entries from ..base_step import BaseStep from ..file_io import extract_daily_date from ...components import R @@ -116,9 +118,12 @@ class SearchV2Step(_ToolContextDedupMixin, BaseStep): expand_links_enabled: bool = bool(self.kwargs.get("expand_links", True)) max_links_per_direction: int = int(self.kwargs.get("max_links_per_direction", 10)) tool_context_id: str = (self.context.get("tool_context_id", "") or "").strip() - strict_date_filter: bool = bool( - self.context.get("strict_date_filter") or self.kwargs.get("strict_date_filter", False), - ) + # Injected value takes precedence over YAML kwargs; check existence + # (not truthiness) so an explicit False can disable a YAML-true flag. + _strict_date_filter = self.context.get("strict_date_filter") + if _strict_date_filter is None: + _strict_date_filter = self.kwargs.get("strict_date_filter", False) + strict_date_filter: bool = bool(_strict_date_filter) if not query: self.context.response.success = False @@ -209,12 +214,15 @@ class SearchV2Step(_ToolContextDedupMixin, BaseStep): ) dialog_dir = self.config_value("dialog_dir") - self.context.response.answer = format_chunks_answer( - fused, + entries = render_chunk_entries( + merge_session_chunk_intervals(fused, dialog_dir), dialog_dir, score_fn=lambda c: self._format_scores(c.scores, hybrid), link_expansion=link_expansion, ) + if self._session_compress_enabled(): + await self._compress_session_entries(entries, query, dialog_dir) + self.context.response.answer = join_chunk_entries(entries) if not fused: self.context.response.answer = ALL_RETURNED_MESSAGE if pre_dedup_count > 0 else NO_RESULTS_MESSAGE self.context.response.metadata["results"] = [ @@ -230,3 +238,93 @@ class SearchV2Step(_ToolContextDedupMixin, BaseStep): if dedup is not None: self.context.response.metadata["dedup"] = dedup return self.context.response + + def _session_compress_enabled(self) -> bool: + """True when the injected ``_search._compress.session`` flag is truthy.""" + assert self.context is not None + search_cfg: dict = self.context.get("_search") or {} + value = (search_cfg.get("_compress") or {}).get("session") + return value is True or str(value).strip().lower() == "true" + + async def _compress_session_entries(self, entries: list[dict[str, str]], query: str, dialog_dir: str) -> None: + """Compress session-transcript entry bodies in place via the ``compressor`` job. + + Only entries whose ``path`` points at a raw session transcript are + compressed; other entries and all non-``body`` fields stay untouched. + When ``_search.type`` is ``query-independent`` the compressor runs + without queries (generic compression); otherwise (``query-aware``, + the default) it receives the injected ``_search.queries`` plus the + current search query. + + The compressor receives the already-rendered body (one message per + line) stripped. Its output is adopted whenever the compressor + succeeded and the result is not longer than the input; adopted bodies + get a leading ``compressed session chunk:`` marker so downstream + consumers can tell them from verbatim transcripts. + + Degrades gracefully when the ``compressor`` job is missing from the + active config: the whole method becomes a no-op and a warning is + logged, so search behaves as if compression were disabled. This + avoids a hard ``Job compressor not found`` failure when a benchmark + config forgets to define the compressor job/component. + + Per-entry exceptions raised by the compressor job (e.g. a temporary + LLM outage) are caught inside ``compress`` so they never propagate + through ``asyncio.gather``: the failing entry keeps its original body + while the remaining entries are still compressed, preserving already + retrieved search results. + """ + assert self.context is not None + # Guard: when the compressor job is missing from the active config + # (e.g. a benchmark config that forgot to define it), degrade + # gracefully to the no-compression behavior instead of raising + # "Job compressor not found" from run_job below. + # Skipped when there is no app_context (e.g. unit tests that mock + # run_job directly), so the mock can still drive compression. + if self.app_context is not None and self.get_job("compressor") is None: + self.logger.warning( + f"[{self.name}] compressor job not found in config; " + "skipping session chunk compression (degrading to uncompressed behavior)", + ) + return + search_cfg: dict = self.context.get("_search") or {} + query_type = str(search_cfg.get("type") or "query-aware").strip().lower() + if query_type == "query-independent": + queries: list[str] = [] + else: + queries = [str(q).strip() for q in (search_cfg.get("queries") or []) if str(q).strip()] + if query and query not in queries: + queries.append(query) + + async def compress(entry: dict[str, str]) -> None: + path = entry.get("path", "") + body = (entry.get("body", "") or "").strip() + if not body: + return + try: + response = await self.run_job("compressor", text=body, queries=queries) + except Exception as exc: # pylint: disable=broad-except + self.logger.warning( + f"[{self.name}] session body compression raised path={path!r} " f"error={exc!r}; keeping original", + ) + return + compressed = str(response.answer or "").strip() + if not response.success or not compressed: + self.logger.warning( + f"[{self.name}] session body compression failed path={path!r} " + f"success={response.success} answer={compressed[:100]!r}; keeping original", + ) + return + if len(compressed) > len(body): + self.logger.info( + f"[{self.name}] compressed body longer than original " + f"({len(compressed)} > {len(body)}) path={path!r}; keeping original", + ) + return + entry["body"] = f"compressed session chunk:\n{compressed}" + + targets = [e for e in entries if is_session_path(e.get("path", ""), dialog_dir)] + if not targets: + return + self.logger.info(f"[{self.name}] compressing {len(targets)} session entries with {len(queries)} queries") + await asyncio.gather(*(compress(entry) for entry in targets)) diff --git a/reme/steps/index/traverse.py b/reme/steps/index/traverse.py index 93d4134b..32de4a6d 100644 --- a/reme/steps/index/traverse.py +++ b/reme/steps/index/traverse.py @@ -1,127 +1,181 @@ -"""BFS over wikilink edges from one or more seed files. - -One record per traversed *edge* (not per node): the same target can repeat -if reached via different anchors or paths. Each record carries the -predecessor and anchor so callers can reconstruct the path. Adjacency is -built once via a single ``file_store.get_nodes()`` call — BFS then runs -purely in memory with no per-frontier round-trips. -""" +"""Build a bounded, frontend-ready graph from indexed Markdown wikilinks.""" from collections import deque +from dataclasses import dataclass from pathlib import Path from ..base_step import BaseStep from ...components import R -from ...schema import FileLink +from ...schema import FileNode, TraverseGraph, TraverseGraphEdge, TraverseGraphNode -_OUT = {"out", "forward", "both"} -_IN = {"in", "backward", "both"} -_VALID = _OUT | _IN - -# source path -> list of (neighbor path, link) -Adjacency = dict[str, list[tuple[str, FileLink]]] +_DIRECTION_ALIASES = { + "out": "forward", + "forward": "forward", + "in": "backward", + "backward": "backward", + "both": "both", +} -async def _build_adjacency(file_store) -> tuple[Adjacency, Adjacency]: - """Single ``get_nodes()`` pass → (outbound, inbound) adjacency maps. +@dataclass(frozen=True, slots=True) +class _TraversalLink: + """An adjacency entry carrying both traversal and original edge direction.""" - Inbound stores the source path next to each link so BFS can attribute - inbound edges back to their origin — ``get_inlinks`` alone returns - target-shaped FileLinks without source attribution. - """ - outbound: Adjacency = {} - inbound: Adjacency = {} - for node in await file_store.get_nodes(): + neighbor: str + source: str + target: str + target_anchor: str | None + + +Adjacency = dict[str, list[_TraversalLink]] +EdgeKey = tuple[str, str, str | None] + + +def _build_adjacency(nodes: list[FileNode]) -> tuple[Adjacency, Adjacency]: + """Build forward/backward adjacency while preserving actual wikilink direction.""" + forward: Adjacency = {} + backward: Adjacency = {} + for node in nodes: for link in node.links: - if link.target_path: - outbound.setdefault(node.path, []).append((link.target_path, link)) - inbound.setdefault(link.target_path, []).append((node.path, link)) - return outbound, inbound + if not link.target_path: + continue + item = _TraversalLink( + neighbor=link.target_path, + source=node.path, + target=link.target_path, + target_anchor=link.target_anchor, + ) + forward.setdefault(node.path, []).append(item) + backward.setdefault(link.target_path, []).append( + _TraversalLink( + neighbor=node.path, + source=item.source, + target=item.target, + target_anchor=item.target_anchor, + ), + ) + return forward, backward -def _bfs( +def _traverse( seeds: list[str], max_depth: int, direction: str, - outbound: Adjacency, - inbound: Adjacency, -) -> list[dict]: - """In-memory BFS; emits one record per unique (src, dst, anchor) edge.""" - sources: list[Adjacency] = [] - if direction in _OUT: - sources.append(outbound) - if direction in _IN: - sources.append(inbound) + forward: Adjacency, + backward: Adjacency, +) -> tuple[dict[str, int], dict[EdgeKey, int]]: + """Return shortest node depths and directed edges reached by bounded BFS.""" + adjacency_maps = [] + if direction in {"forward", "both"}: + adjacency_maps.append(forward) + if direction in {"backward", "both"}: + adjacency_maps.append(backward) - visited: set[tuple[str, str, str | None]] = set() - results: list[dict] = [] - queue: deque[tuple[str, int]] = deque((s, 0) for s in seeds) + node_depths = {seed: 0 for seed in seeds} + edge_depths: dict[EdgeKey, int] = {} + queue = deque(seeds) while queue: - current, depth = queue.popleft() - if depth >= max_depth: + current = queue.popleft() + current_depth = node_depths[current] + if current_depth >= max_depth: continue - for src in sources: - for next_path, link in src.get(current, ()): - key = (current, next_path, link.target_anchor) - if key in visited: - continue - visited.add(key) - results.append( - { - "path": next_path, - "depth": depth + 1, - "via": current, - "anchor": link.target_anchor, - }, - ) - if depth + 1 < max_depth: - queue.append((next_path, depth + 1)) - return results + + next_depth = current_depth + 1 + for adjacency in adjacency_maps: + for link in adjacency.get(current, ()): + edge_key = (link.source, link.target, link.target_anchor) + previous_edge_depth = edge_depths.get(edge_key) + if previous_edge_depth is None or next_depth < previous_edge_depth: + edge_depths[edge_key] = next_depth + + previous_node_depth = node_depths.get(link.neighbor) + if previous_node_depth is None or next_depth < previous_node_depth: + node_depths[link.neighbor] = next_depth + queue.append(link.neighbor) + + return node_depths, edge_depths + + +def _build_graph( + *, + seeds: list[str], + max_depth: int, + direction: str, + indexed_nodes: list[FileNode], + node_depths: dict[str, int], + edge_depths: dict[EdgeKey, int], +) -> TraverseGraph: + """Materialize deterministic public graph schemas from traversal state.""" + node_by_path = {node.path: node for node in indexed_nodes} + graph_nodes = [] + for path, node_depth in sorted(node_depths.items(), key=lambda item: (item[1], item[0])): + node = node_by_path.get(path) + graph_nodes.append( + TraverseGraphNode( + id=path, + path=path, + name=node.front_matter.name if node is not None else "", + description=node.front_matter.description if node is not None else "", + depth=node_depth, + indexed=node is not None, + ), + ) + + graph_edges = [ + TraverseGraphEdge(source=source, target=target, target_anchor=anchor, depth=edge_depth) + for (source, target, anchor), edge_depth in sorted( + edge_depths.items(), + key=lambda item: (item[1], item[0][0], item[0][1], item[0][2] or ""), + ) + ] + return TraverseGraph( + seeds=seeds, + depth=max_depth, + direction=direction, + nodes=graph_nodes, + edges=graph_edges, + ) @R.register("traverse_step") class TraverseStep(BaseStep): - """BFS from one or more seed files to explore wikilink relationships. - - Parameters: - path — single seed (str) or list of seeds (workspace-relative). - direction — ``forward`` / ``backward`` / ``both`` (or ``out`` / ``in`` / ``both``). - depth — hop limit (default 1 = immediate neighbors). - """ + """Return a bounded wikilink graph rooted at one or more workspace paths.""" async def execute(self): assert self.context is not None - raw = self.context.get("path") - items = [raw] if isinstance(raw, (str, Path)) else list(raw or []) - seeds = [str(p) for p in items if p] - assert seeds, "path is required" - depth = int(self.context.get("depth") or 1) - direction = (self.context.get("direction") or "both").lower() - assert direction in _VALID, f"direction must be one of {sorted(_VALID)}, got {direction!r}" + raw_paths = self.context.get("path") + items = [raw_paths] if isinstance(raw_paths, (str, Path)) else list(raw_paths or []) + seeds = list(dict.fromkeys(str(path).replace("\\", "/") for path in items if path)) + if not seeds: + raise ValueError("path is required") - outbound, inbound = await _build_adjacency(self.file_store) - results = _bfs(seeds, depth, direction, outbound, inbound) + raw_depth = self.context.get("depth") + max_depth = 1 if raw_depth is None else int(raw_depth) + if max_depth < 0: + raise ValueError("depth must be greater than or equal to 0") - self.logger.info( - f"[{self.name}] seeds={seeds!r} depth={depth} direction={direction} " - f"nodes={len(outbound) + len(inbound)} edges={len(results)}", + raw_direction = str(self.context.get("direction") or "both").lower() + direction = _DIRECTION_ALIASES.get(raw_direction) + if direction is None: + raise ValueError(f"direction must be one of {sorted(_DIRECTION_ALIASES)}, got {raw_direction!r}") + + indexed_nodes = await self.file_store.get_nodes() + forward, backward = _build_adjacency(indexed_nodes) + node_depths, edge_depths = _traverse(seeds, max_depth, direction, forward, backward) + graph = _build_graph( + seeds=seeds, + max_depth=max_depth, + direction=direction, + indexed_nodes=indexed_nodes, + node_depths=node_depths, + edge_depths=edge_depths, ) - label = seeds[0] if len(seeds) == 1 else f"{len(seeds)} seeds" - if not results: - answer = f"No edges found from {label}" - else: - header = f"Traversed {len(results)} edge(s) from {label}" - lines = [header, ""] - for r in results: - target = r["path"] - if r["anchor"]: - target = f"{target}#{r['anchor']}" - lines.append(f"[depth={r['depth']}] {r['via']} --> {target}") - answer = "\n".join(lines) - self.context.response.success = True - self.context.response.answer = answer - self.context.response.metadata.update({"edges": results, "count": len(results)}) + self.context.response.answer = graph.model_dump() + self.logger.info( + f"[{self.name}] seeds={seeds!r} depth={max_depth} direction={direction} " + f"nodes={len(graph.nodes)} edges={len(graph.edges)}", + ) return self.context.response diff --git a/reme/steps/index/vector_search.py b/reme/steps/index/vector_search.py index e6d103c7..c1c11f42 100644 --- a/reme/steps/index/vector_search.py +++ b/reme/steps/index/vector_search.py @@ -3,7 +3,8 @@ from typing import Final from ._dedup import _ToolContextDedupMixin -from ._source_format import ALL_RETURNED_MESSAGE, NO_RESULTS_MESSAGE, format_chunks_answer +from ._source_format import ALL_RETURNED_MESSAGE, NO_RESULTS_MESSAGE, join_chunk_entries +from ._source_format import merge_session_chunk_intervals, render_chunk_entries from ..base_step import BaseStep from ...components import R @@ -48,11 +49,12 @@ class VectorSearchStep(_ToolContextDedupMixin, BaseStep): results = results[:limit] dialog_dir = self.config_value("dialog_dir") - self.context.response.answer = format_chunks_answer( - results, + entries = render_chunk_entries( + merge_session_chunk_intervals(results, dialog_dir), dialog_dir, include_source=self.include_source, ) + self.context.response.answer = join_chunk_entries(entries) if not results: self.context.response.answer = ALL_RETURNED_MESSAGE if pre_dedup_count > 0 else NO_RESULTS_MESSAGE self.context.response.metadata["results"] = [ diff --git a/scripts/github_star_growth.py b/scripts/github_star_growth.py new file mode 100755 index 00000000..b398d128 --- /dev/null +++ b/scripts/github_star_growth.py @@ -0,0 +1,281 @@ +#!/usr/bin/env python3 +"""Report daily new GitHub stars for a repository using the GitHub CLI.""" + +from __future__ import annotations + +import argparse +import csv +import json +import math +import subprocess +import sys +from collections import Counter +from datetime import UTC, date, datetime, timedelta +from pathlib import Path +from typing import Any, TextIO +from xml.sax.saxutils import escape + +DEFAULT_REPOSITORY = "agentscope-ai/ReMe" +QUERY = """ +query($owner: String!, $name: String!, $before: String) { + repository(owner: $owner, name: $name) { + nameWithOwner + stargazerCount + stargazers(last: 100, before: $before) { + edges { + starredAt + } + pageInfo { + hasPreviousPage + startCursor + } + } + } +} +""" + + +def parse_args() -> argparse.Namespace: + """Parse and validate command-line arguments.""" + parser = argparse.ArgumentParser( + description="Print daily new-star counts for the latest N UTC calendar days.", + ) + parser.add_argument( + "--repo", + default=DEFAULT_REPOSITORY, + metavar="OWNER/REPO", + help=f"GitHub repository (default: {DEFAULT_REPOSITORY})", + ) + parser.add_argument( + "--days", + type=int, + default=365, + help="number of UTC calendar days to include (default: 365)", + ) + parser.add_argument( + "--output", + type=Path, + metavar="PATH", + help="write CSV to PATH instead of standard output", + ) + parser.add_argument( + "--chart", + type=Path, + metavar="PATH", + help="write the SVG chart to PATH (default: CSV path with an .svg suffix)", + ) + args = parser.parse_args() + + if args.days < 1: + parser.error("--days must be at least 1") + if args.repo.count("/") != 1 or any(not part for part in args.repo.split("/")): + parser.error("--repo must have the form OWNER/REPO") + return args + + +def query_github(owner: str, name: str, before: str | None) -> dict[str, Any]: + """Fetch one page of repository stargazers through ``gh api graphql``.""" + request = { + "query": QUERY, + "variables": {"owner": owner, "name": name, "before": before}, + } + try: + result = subprocess.run( + ["gh", "api", "graphql", "--input", "-"], + input=json.dumps(request), + text=True, + capture_output=True, + check=False, + ) + except FileNotFoundError as exc: + raise RuntimeError("GitHub CLI 'gh' is not installed or is not on PATH") from exc + + if result.returncode != 0: + detail = result.stderr.strip() or result.stdout.strip() + raise RuntimeError(f"gh api graphql failed: {detail}") + + try: + payload = json.loads(result.stdout) + except json.JSONDecodeError as exc: + raise RuntimeError("gh returned invalid JSON") from exc + + if payload.get("errors"): + messages = "; ".join(error.get("message", str(error)) for error in payload["errors"]) + raise RuntimeError(f"GitHub GraphQL API error: {messages}") + + repository = payload.get("data", {}).get("repository") + if repository is None: + raise RuntimeError(f"repository not found or inaccessible: {owner}/{name}") + return repository + + +def fetch_daily_stars(repository: str, start_date: date) -> tuple[Counter[date], int]: + """Fetch current stargazers and group their star timestamps by UTC date.""" + owner, name = repository.split("/", maxsplit=1) + daily_stars: Counter[date] = Counter() + before: str | None = None + total_stars = 0 + + while True: + data = query_github(owner, name, before) + total_stars = data["stargazerCount"] + connection = data["stargazers"] + edges = connection["edges"] + + reached_start = False + for edge in edges: + starred_at = datetime.fromisoformat(edge["starredAt"].replace("Z", "+00:00")) + starred_date = starred_at.astimezone(UTC).date() + if starred_date < start_date: + reached_start = True + continue + daily_stars[starred_date] += 1 + + page_info = connection["pageInfo"] + if reached_start or not page_info["hasPreviousPage"]: + break + before = page_info["startCursor"] + if before is None: + raise RuntimeError("GitHub returned an invalid pagination cursor") + + return daily_stars, total_stars + + +def write_csv(stream: TextIO, start_date: date, days: int, counts: Counter[date]) -> None: + """Write a row for every date in the requested period, including zero days.""" + writer = csv.writer(stream, lineterminator="\n") + writer.writerow(["date", "new_stars"]) + for offset in range(days): + current_date = start_date + timedelta(days=offset) + writer.writerow([current_date.isoformat(), counts[current_date]]) + + +def nice_tick_step(maximum: int, tick_count: int = 5) -> int: + """Return a readable positive interval for numeric axis ticks.""" + rough_step = max(maximum, 1) / tick_count + magnitude = 10 ** math.floor(math.log10(rough_step)) + normalized = rough_step / magnitude + if normalized <= 1: + multiplier = 1 + elif normalized <= 2: + multiplier = 2 + elif normalized <= 5: + multiplier = 5 + else: + multiplier = 10 + return max(1, multiplier * magnitude) + + +def write_svg_chart(path: Path, repository: str, dates: list[date], values: list[int]) -> None: + """Render daily star increments as a dependency-free SVG bar chart.""" + width, height = 1280, 640 + margin_left, margin_right = 75, 30 + margin_top, margin_bottom = 70, 85 + plot_width = width - margin_left - margin_right + plot_height = height - margin_top - margin_bottom + + tick_step = nice_tick_step(max(values, default=0)) + axis_max = max(tick_step, math.ceil(max(values, default=0) / tick_step) * tick_step) + bar_slot = plot_width / len(values) + bar_width = max(0.8, bar_slot * 0.82) + + svg = [ + '', + ( + f'' + ), + f"{escape(repository)} daily new GitHub stars", + '', + ( + f'{escape(repository)} daily new stars' + ), + ] + + for tick in range(0, axis_max + 1, tick_step): + y = margin_top + plot_height - (tick / axis_max * plot_height) + svg.append( + f'', + ) + svg.append( + f'{tick}', + ) + + for index, (current_date, value) in enumerate(zip(dates, values, strict=True)): + bar_height = value / axis_max * plot_height + x = margin_left + index * bar_slot + (bar_slot - bar_width) / 2 + y = margin_top + plot_height - bar_height + svg.append( + f'{current_date.isoformat()}: {value} new stars', + ) + + label_count = min(12, len(dates)) + label_indexes = sorted({round(index * (len(dates) - 1) / max(label_count - 1, 1)) for index in range(label_count)}) + baseline = margin_top + plot_height + for index in label_indexes: + x = margin_left + (index + 0.5) * bar_slot + svg.append( + f'', + ) + svg.append( + f'{dates[index].isoformat()}', + ) + + svg.extend( + [ + ( + f'' + ), + ( + f'New stars' + ), + "", + ], + ) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("\n".join(svg) + "\n", encoding="utf-8") + + +def main() -> int: + """Run the report and return a process exit code.""" + args = parse_args() + end_date = datetime.now(UTC).date() + start_date = end_date - timedelta(days=args.days - 1) + + try: + counts, total_stars = fetch_daily_stars(args.repo, start_date) + dates = [start_date + timedelta(days=offset) for offset in range(args.days)] + values = [counts[current_date] for current_date in dates] + if args.output: + args.output.parent.mkdir(parents=True, exist_ok=True) + with args.output.open("w", encoding="utf-8", newline="") as stream: + write_csv(stream, start_date, args.days, counts) + else: + write_csv(sys.stdout, start_date, args.days, counts) + chart_path = args.chart or (args.output.with_suffix(".svg") if args.output else Path("star_growth.svg")) + write_svg_chart(chart_path, args.repo, dates, values) + except (OSError, RuntimeError) as exc: + print(f"error: {exc}", file=sys.stderr) + return 1 + + destination = str(args.output) if args.output else "stdout" + period_stars = sum(counts.values()) + print( + f"Fetched {period_stars} current stargazers in {start_date}..{end_date}; " + f"repository currently has {total_stars} stars. CSV: {destination}; chart: {chart_path}", + file=sys.stderr, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/unit/test_app_config_step.py b/tests/unit/test_app_config_step.py new file mode 100644 index 00000000..afbc2600 --- /dev/null +++ b/tests/unit/test_app_config_step.py @@ -0,0 +1,44 @@ +"""Tests for the application config step.""" + +import asyncio + +from reme.components.application_context import ApplicationContext +from reme.steps.common.app_config import AppConfigStep + + +def test_app_config_step_returns_effective_config_without_secrets(tmp_path): + """Effective values are returned while startup secrets stay private.""" + context = ApplicationContext( + app_name="Test ReMe", + workspace_dir=str(tmp_path), + environment={"PRIVATE_VALUE": "hidden"}, + components={ + "as_llm": { + "default": { + "backend": "openai", + "credential": {"api_key": "key", "base_url": "https://example.test"}, + }, + }, + }, + mcp_servers={ + "private": { + "url": "https://user:password@example.test/mcp?access_token=url-token&mode=read", + "headers": { + "Authorization": "Bearer header-token", + "X-API-Key": "header-key", + }, + "client_secret": "client-secret", + }, + }, + ) + + response = asyncio.run(AppConfigStep(app_context=context)()) + + assert response.answer["app_name"] == "Test ReMe" + assert "environment" not in response.answer + credential = response.answer["components"]["as_llm"]["default"]["credential"] + assert credential == {"api_key": "***", "base_url": "https://example.test"} + mcp_server = response.answer["mcp_servers"]["private"] + assert mcp_server["url"] == "https://user:***@example.test/mcp?access_token=%2A%2A%2A&mode=read" + assert mcp_server["headers"] == {"Authorization": "***", "X-API-Key": "***"} + assert mcp_server["client_secret"] == "***" diff --git a/tests/unit/test_beam_auto_memory.py b/tests/unit/test_beam_auto_memory.py new file mode 100644 index 00000000..bddb3e79 --- /dev/null +++ b/tests/unit/test_beam_auto_memory.py @@ -0,0 +1,128 @@ +"""Unit tests for BEAM auto-memory segmented ingestion and line numbering.""" + +# pylint: disable=missing-class-docstring,missing-function-docstring,protected-access + +from agentscope.message import Msg + +from reme.components.runtime_context import RuntimeContext +from reme.steps.benchmark.beam.auto_memory import ( + BeamAutoMemoryStep, + split_turn_segments, +) + + +def _msg(role: str, words: int, created_at: str = "2024-03-01T09:00:00") -> Msg: + return Msg( + name=role, + role=role, + content=[{"type": "text", "text": " ".join(["w"] * words)}], + created_at=created_at, + ) + + +def _dialog(n_turns: int, words_per_msg: int) -> list[Msg]: + messages: list[Msg] = [] + for _ in range(n_turns): + messages.append(_msg("user", words_per_msg)) + messages.append(_msg("assistant", words_per_msg)) + return messages + + +class TestSplitTurnSegments: + def test_empty(self): + assert not split_turn_segments([], 100) + + def test_disabled_returns_single_segment(self): + messages = _dialog(3, 10) + segments = split_turn_segments(messages, 0) + assert len(segments) == 1 + assert segments[0] == (0, messages) + + def test_under_limit_single_segment(self): + messages = _dialog(3, 10) # 60 words total + segments = split_turn_segments(messages, 100) + assert len(segments) == 1 + assert segments[0][0] == 0 + assert segments[0][1] == messages + + def test_splits_at_turn_boundaries(self): + # 4 turns x 20 words each; limit 40 -> 2 turns per segment + messages = _dialog(4, 10) + segments = split_turn_segments(messages, 40) + assert len(segments) == 2 + offsets = [offset for offset, _ in segments] + assert offsets == [0, 4] + # Every segment starts with a user message and ends with an assistant + for _, segment in segments: + assert segment[0].role == "user" + assert segment[-1].role == "assistant" + # No message lost or duplicated, order preserved + flattened = [m for _, segment in segments for m in segment] + assert flattened == messages + + def test_never_splits_inside_a_turn(self): + # One turn alone exceeds the limit -> becomes its own oversized segment + messages = [ + _msg("user", 5), + _msg("assistant", 5), + _msg("user", 50), + _msg("assistant", 50), # 100-word turn > limit 60 + _msg("user", 5), + _msg("assistant", 5), + ] + segments = split_turn_segments(messages, 60) + assert [offset for offset, _ in segments] == [0, 2, 4] + assert [len(segment) for _, segment in segments] == [2, 2, 2] + + def test_offsets_are_original_indices(self): + messages = _dialog(5, 30) # 60 words per turn + segments = split_turn_segments(messages, 120) + # 2 turns per segment -> offsets 0, 4, 8 + assert [offset for offset, _ in segments] == [0, 4, 8] + for offset, segment in segments: + for i, msg in enumerate(segment): + assert msg is messages[offset + i] + + def test_multi_assistant_turn_stays_together(self): + messages = [ + _msg("user", 10), + _msg("assistant", 10), + _msg("assistant", 10), + _msg("user", 10), + _msg("assistant", 10), + ] + segments = split_turn_segments(messages, 30) + assert [offset for offset, _ in segments] == [0, 3] + assert len(segments[0][1]) == 3 + + +class TestFormatHistoryLineNumbers: + def _step(self, offset: int) -> BeamAutoMemoryStep: + step = BeamAutoMemoryStep(name="beam_auto_memory_step", backend="beam_auto_memory_step") + step.context = RuntimeContext( + session_id="beam_1M_1_batch1", + beam_line_offset=offset, + ) + return step + + def test_numbers_start_at_one_without_offset(self): + step = self._step(0) + history = step._format_history(_dialog(2, 3)) + assert "[L1 | user @" in history + assert "[L4 | assistant @" in history + assert "lines 1-4" in history + assert "session/dialog/beam_1M_1_batch1.jsonl" in history + + def test_numbers_use_original_file_offset(self): + step = self._step(40) + history = step._format_history(_dialog(2, 3)) + assert "[L41 | user @" in history + assert "[L44 | assistant @" in history + assert "lines 41-44" in history + assert "[L1 " not in history + + def test_build_messages_passes_msg_objects_through(self): + step = self._step(0) + messages = _dialog(2, 3) + rebuilt = step._build_messages(messages) + assert [m.id for m in rebuilt] == [m.id for m in messages] diff --git a/tests/unit/test_common_steps.py b/tests/unit/test_common_steps.py index 600ee2ca..1c6aa63d 100644 --- a/tests/unit/test_common_steps.py +++ b/tests/unit/test_common_steps.py @@ -10,7 +10,7 @@ import warnings from reme.components.agent_wrapper import BaseAgentWrapper from reme.components.application_context import ApplicationContext from reme.components.file_store import LocalFileStore -from reme.schema import FileLink, FileNode +from reme.schema import FileFrontMatter, FileLink, FileNode, TraverseGraph from reme.steps.common.add import AddStep from reme.steps.common.health_check import _file_graph_status from reme.steps.common.llm_demo import LLMDemoStep @@ -42,7 +42,13 @@ def _run(coro): asyncio.run(coro) -def _node(path: str, links: list[tuple[str, str | None]] | None = None) -> FileNode: +def _node( + path: str, + links: list[tuple[str, str | None]] | None = None, + *, + name: str = "", + description: str = "", +) -> FileNode: """Build a FileNode with (target_path, target_anchor) outgoing edges.""" return FileNode( path=path, @@ -50,6 +56,7 @@ def _node(path: str, links: list[tuple[str, str | None]] | None = None) -> FileN links=[ FileLink(source_path=path, target_path=target, target_anchor=anchor) for target, anchor in (links or []) ], + front_matter=FileFrontMatter(name=name, description=description), ) @@ -63,7 +70,7 @@ async def _make_store(nodes: list[FileNode]) -> LocalFileStore: def _edges(step) -> list[dict]: - return step.context.response.metadata.get("edges", []) + return step.context.response.answer["edges"] def test_add_step_coerces_numeric_inputs(): @@ -207,12 +214,11 @@ def test_traverse_forward_depth_1(): step = traverse_mod.TraverseStep(file_store=store) await step(path="a.md", direction="forward", depth=1) results = _edges(step) - paths = {r["path"] for r in results} + paths = {r["target"] for r in results} assert paths == {"b.md", "c.md"} - # Anchors remain part of traversal metadata. - c_edge = next(r for r in results if r["path"] == "c.md") - assert c_edge["anchor"] == "intro" - assert c_edge["via"] == "a.md" + c_edge = next(r for r in results if r["target"] == "c.md") + assert c_edge["target_anchor"] == "intro" + assert c_edge["source"] == "a.md" assert c_edge["depth"] == 1 await store.close() print("✓ test_traverse_forward_depth_1 passed") @@ -220,6 +226,25 @@ def test_traverse_forward_depth_1(): asyncio.run(run()) +def test_traverse_normalizes_windows_seed_path(): + """Windows-style seeds match the graph's portable POSIX path keys.""" + + async def run(): + with tempfile.TemporaryDirectory() as tmp, _temp_chdir(tmp): + store = await _make_store( + [ + _node("topics/a.md", [("topics/b.md", None)]), + _node("topics/b.md"), + ], + ) + step = traverse_mod.TraverseStep(file_store=store) + await step(path=r"topics\a.md", direction="forward", depth=1) + assert [edge["target"] for edge in _edges(step)] == ["topics/b.md"] + await store.close() + + asyncio.run(run()) + + def test_traverse_backward_returns_inlinks(): """direction=backward walks inbound edges.""" @@ -235,7 +260,10 @@ def test_traverse_backward_returns_inlinks(): step = traverse_mod.TraverseStep(file_store=store) await step(path="b.md", direction="backward", depth=1) results = _edges(step) - assert {r["path"] for r in results} == {"a.md", "c.md"} + assert {(r["source"], r["target"]) for r in results} == { + ("a.md", "b.md"), + ("c.md", "b.md"), + } await store.close() print("✓ test_traverse_backward_returns_inlinks passed") @@ -257,7 +285,7 @@ def test_traverse_depth_2_expands(): step = traverse_mod.TraverseStep(file_store=store) await step(path="a.md", direction="forward", depth=2) results = _edges(step) - depth_map = {r["path"]: r["depth"] for r in results} + depth_map = {r["target"]: r["depth"] for r in results} assert depth_map.get("b.md") == 1 assert depth_map.get("c.md") == 2 await store.close() @@ -320,13 +348,96 @@ def test_traverse_both_directions(): step = traverse_mod.TraverseStep(file_store=store) await step(path="center.md", direction="both", depth=1) results = _edges(step) - assert {r["path"] for r in results} == {"upstream.md", "downstream.md"} + assert {(r["source"], r["target"]) for r in results} == { + ("upstream.md", "center.md"), + ("center.md", "downstream.md"), + } await store.close() print("✓ test_traverse_both_directions passed") asyncio.run(run()) +def test_traverse_both_preserves_reciprocal_edge_directions(): + """Opposite wikilinks remain two distinct directed graph edges.""" + + async def run(): + with tempfile.TemporaryDirectory() as tmp, _temp_chdir(tmp): + store = await _make_store( + [ + _node("a.md", [("b.md", None)]), + _node("b.md", [("a.md", None)]), + ], + ) + step = traverse_mod.TraverseStep(file_store=store) + response = await step(path="a.md", direction="both", depth=1) + graph = TraverseGraph.model_validate(response.answer) + + assert {(edge.source, edge.target) for edge in graph.edges} == { + ("a.md", "b.md"), + ("b.md", "a.md"), + } + assert response.metadata == {} + await store.close() + + asyncio.run(run()) + + +def test_traverse_returns_frontmatter_and_unresolved_nodes(): + """Graph nodes expose labels and distinguish indexed files from dangling targets.""" + + async def run(): + with tempfile.TemporaryDirectory() as tmp, _temp_chdir(tmp): + store = await _make_store( + [ + _node( + "a.md", + [("missing.md", "details")], + name="Alpha", + description="Root note", + ), + ], + ) + step = traverse_mod.TraverseStep(file_store=store) + response = await step(path="a.md", direction="forward", depth=1) + graph = TraverseGraph.model_validate(response.answer) + nodes = {node.path: node for node in graph.nodes} + + assert graph.version == 1 + assert graph.seeds == ["a.md"] + assert nodes["a.md"].model_dump() == { + "id": "a.md", + "path": "a.md", + "name": "Alpha", + "description": "Root note", + "depth": 0, + "indexed": True, + } + assert nodes["missing.md"].indexed is False + assert nodes["missing.md"].depth == 1 + assert graph.edges[0].target_anchor == "details" + await store.close() + + asyncio.run(run()) + + +def test_traverse_depth_zero_returns_only_seed_nodes(): + """A zero-hop traversal is valid and emits no edges.""" + + async def run(): + with tempfile.TemporaryDirectory() as tmp, _temp_chdir(tmp): + store = await _make_store([_node("a.md", [("b.md", None)]), _node("b.md")]) + step = traverse_mod.TraverseStep(file_store=store) + response = await step(path="a.md", direction="forward", depth=0) + graph = TraverseGraph.model_validate(response.answer) + + assert [node.path for node in graph.nodes] == ["a.md"] + assert graph.edges == [] + await store.close() + + asyncio.run(run()) + + if __name__ == "__main__": print("\n=== traverse step tests ===") test_traverse_forward_depth_1() @@ -335,4 +446,7 @@ if __name__ == "__main__": test_traverse_short_seed_yields_empty() test_traverse_not_found_seed() test_traverse_both_directions() + test_traverse_both_preserves_reciprocal_edge_directions() + test_traverse_returns_frontmatter_and_unresolved_nodes() + test_traverse_depth_zero_returns_only_seed_nodes() print("\n所有测试通过!") diff --git a/tests/unit/test_compressor_step.py b/tests/unit/test_compressor_step.py new file mode 100644 index 00000000..b2cbf721 --- /dev/null +++ b/tests/unit/test_compressor_step.py @@ -0,0 +1,72 @@ +"""Unit tests for CompressorStep without LLM dependencies.""" + +import asyncio + +from agentscope.message import TextBlock, ThinkingBlock +from agentscope.model import ChatResponse + +from reme.steps.evolve.compressor import CompressorStep + + +def test_response_text_handles_non_streaming_chat_response(): + """A plain ChatResponse must not crash the async-iteration check. + + Regression: ``ChatResponse`` inherits agentscope's ``DictMixin`` whose + instance ``__getattr__`` raises ``KeyError``, so ``hasattr(result, + "__aiter__")`` raised instead of returning False and every compression + failed with ``KeyError: '__aiter__'``. + """ + + async def run(): + result = ChatResponse( + content=[ + {"type": "thinking", "thinking": "internal reasoning"}, + {"type": "text", "text": "compressed "}, + {"type": "text", "text": "output"}, + ], + is_last=True, + ) + + text = await CompressorStep._response_text(result) # pylint: disable=protected-access + + assert text == "compressed output" + + asyncio.run(run()) + + +def test_response_text_handles_pydantic_content_blocks(): + """Real LLM responses carry pydantic block models, not dicts. + + Regression: ``block.get(...)`` crashed with ``'ThinkingBlock' object has + no attribute 'get'`` on responses from reasoning models. + """ + + async def run(): + result = ChatResponse( + content=[ + ThinkingBlock(type="thinking", thinking="internal reasoning"), + TextBlock(type="text", text="compressed output"), + ], + is_last=True, + ) + + text = await CompressorStep._response_text(result) # pylint: disable=protected-access + + assert text == "compressed output" + + asyncio.run(run()) + + +def test_response_text_consumes_streaming_chunks_and_keeps_last(): + """An async generator of ChatResponses is drained and the last chunk wins.""" + + async def run(): + async def stream(): + yield ChatResponse(content=[{"type": "text", "text": "partial"}], is_last=False) + yield ChatResponse(content=[{"type": "text", "text": "final text"}], is_last=True) + + text = await CompressorStep._response_text(stream()) # pylint: disable=protected-access + + assert text == "final text" + + asyncio.run(run()) diff --git a/tests/unit/test_config_parser.py b/tests/unit/test_config_parser.py index 607389d2..8ea287b5 100644 --- a/tests/unit/test_config_parser.py +++ b/tests/unit/test_config_parser.py @@ -49,6 +49,24 @@ def test_default_config_registers_daily_write_job(): assert job["parameters"]["required"] == ["name", "description", "session_id", "content"] +def test_default_config_registers_app_config_job(): + """The default service exposes the effective application config.""" + job = _load_config("default.yaml")["jobs"]["app_config"] + + assert job["backend"] == "base" + assert job["steps"] == [{"backend": "app_config_step"}] + + +def test_default_config_registers_workspace_web_jobs(): + """The local web client has stable save and streaming chat actions.""" + jobs = _load_config("default.yaml")["jobs"] + + assert jobs["save"]["steps"] == [{"backend": "save_step"}] + assert jobs["load"]["steps"] == [{"backend": "load_step", "max_bytes": 5242880}] + assert jobs["chat"]["backend"] == "stream" + assert jobs["chat"]["steps"] == [{"backend": "chat_step", "agent_wrapper": "default"}] + + def test_default_config_keeps_frontmatter_chunk_metadata_opt_in(): """Markdown frontmatter-to-chunk metadata is disabled by default for compatibility.""" cfg = _load_config("default.yaml") diff --git a/tests/unit/test_crud_steps.py b/tests/unit/test_crud_steps.py index 2fcbf676..1132a686 100644 --- a/tests/unit/test_crud_steps.py +++ b/tests/unit/test_crud_steps.py @@ -30,7 +30,7 @@ import asyncio import os import tempfile import warnings -from pathlib import Path +from pathlib import Path, PureWindowsPath from reme.components.file_store import LocalFileStore from reme.schema import FileNode @@ -177,6 +177,14 @@ def test_list_lists_files(): asyncio.run(run()) +def test_list_formats_workspace_relative_windows_paths_as_posix(): + """Workspace-relative list results use the graph's portable path format.""" + workspace = PureWindowsPath("C:/workspace") + files = [workspace / "topics" / "a.md"] + + assert crud_list.ListStep._format_relative(files, workspace) == ["topics/a.md"] + + def test_list_empty_directory_has_explicit_answer(): """list tells an LLM explicitly when the target directory has no files.""" @@ -770,6 +778,39 @@ def test_read_empty_path_rejected(): _run(run()) +def test_read_format_session_injected_false_disables_yaml_true(): + """Runtime-injected ``read_step_format_session=False`` takes precedence + over a YAML-true config — existence check, not truthiness.""" + + async def run(): + from agentscope.message import Msg + + m = Msg(name="user", role="user", content=[{"type": "text", "text": "hello world"}]) + jsonl_body = m.model_dump_json() + "\n" + + with tempfile.TemporaryDirectory() as tmp, temp_chdir(tmp): + session_path = Path(tmp) / "session" / "dialog" / "s.jsonl" + session_path.parent.mkdir(parents=True, exist_ok=True) + session_path.write_text(jsonl_body, encoding="utf-8") + + store = await _make_store() + + # YAML config read_step_format_session=True, no injection → formatting active. + step = crud_read.ReadStep(file_store=store, read_step_format_session=True) + await step(path="session/dialog/s.jsonl") + assert "[user @" in str(step.context.response.answer) + + # Same YAML config, but runtime injects False → must disable formatting. + step = crud_read.ReadStep(file_store=store, read_step_format_session=True) + await step(path="session/dialog/s.jsonl", read_step_format_session=False) + assert "[user @" not in str(step.context.response.answer) + + await store.close() + print("✓ test_read_format_session_injected_false_disables_yaml_true passed") + + _run(run()) + + # -- write / edit -------------------------------------------------------- diff --git a/tests/unit/test_graph_snapshot_step.py b/tests/unit/test_graph_snapshot_step.py new file mode 100644 index 00000000..c803abcf --- /dev/null +++ b/tests/unit/test_graph_snapshot_step.py @@ -0,0 +1,94 @@ +"""Tests for the frontend-ready category-rooted digest graph snapshot.""" + +import asyncio + +from reme.components.file_store import LocalFileStore +from reme.schema import FileFrontMatter, FileLink, FileNode, GraphSnapshot +from reme.steps.index.graph_snapshot import GraphSnapshotStep + + +def _node( + path: str, + links: list[tuple[str, str | None]] | None = None, + *, + name: str = "", + description: str = "", +) -> FileNode: + return FileNode( + path=path, + st_mtime=1.0, + links=[ + FileLink(source_path=path, target_path=target, target_anchor=anchor) for target, anchor in (links or []) + ], + front_matter=FileFrontMatter(name=name, description=description), + ) + + +def test_graph_snapshot_returns_category_roots_digest_links_and_daily_leaves(tmp_path, monkeypatch): + """The snapshot is rooted at digest categories and stops at indexed daily notes.""" + monkeypatch.chdir(tmp_path) + + async def run(): + store = LocalFileStore(name="snapshot", embedding_store="") + await store.start() + try: + await store.file_graph.upsert_nodes( + [ + _node( + "digest/wiki/alpha.md", + [ + ("digest/personal/beta.md", "intro"), + ("daily/2026-08-05/event.md", None), + ("daily/2026-08-05/missing.md", None), + ("resource/ignored.md", None), + ], + name="Alpha", + description="Root note", + ), + _node("digest/personal/beta.md", [("digest/wiki/alpha.md", None)], name="Beta"), + _node("digest/procedure/how-to.md", name="How to"), + _node( + "daily/2026-08-05/event.md", + [("digest/procedure/how-to.md", None), ("daily/2026-08-05/hidden.md", None)], + name="Event", + ), + _node("daily/2026-08-05/isolated.md", name="Isolated daily"), + _node("resource/ignored.md", name="Ignored resource"), + ], + ) + + response = await GraphSnapshotStep(file_store=store)() + graph = GraphSnapshot.model_validate(response.answer) + + assert response.success is True + assert response.metadata == {} + assert [node.id for node in graph.nodes[:3]] == [ + "virtual:wiki", + "virtual:personal", + "virtual:procedure", + ] + assert [node.path for node in graph.nodes[3:]] == [ + "daily/2026-08-05/event.md", + "digest/personal/beta.md", + "digest/procedure/how-to.md", + "digest/wiki/alpha.md", + ] + nodes = {node.id: node for node in graph.nodes} + assert nodes["virtual:wiki"].path == "digest/wiki" + assert nodes["virtual:procedure"].path == "digest/procedure" + assert nodes["virtual:wiki"].virtual is True + assert nodes["digest/wiki/alpha.md"].name == "Alpha" + assert nodes["digest/wiki/alpha.md"].description == "Root note" + assert nodes["digest/wiki/alpha.md"].virtual is False + assert {(edge.source, edge.target, edge.target_anchor) for edge in graph.edges} == { + ("virtual:wiki", "digest/wiki/alpha.md", None), + ("virtual:personal", "digest/personal/beta.md", None), + ("virtual:procedure", "digest/procedure/how-to.md", None), + ("digest/wiki/alpha.md", "digest/personal/beta.md", "intro"), + ("digest/wiki/alpha.md", "daily/2026-08-05/event.md", None), + ("digest/personal/beta.md", "digest/wiki/alpha.md", None), + } + finally: + await store.close() + + asyncio.run(run()) diff --git a/tests/unit/test_search_step.py b/tests/unit/test_search_step.py index 59bfa6c0..cfa9d575 100644 --- a/tests/unit/test_search_step.py +++ b/tests/unit/test_search_step.py @@ -2,11 +2,13 @@ import asyncio +from agentscope.message import Msg + from reme.components.file_store import BaseFileStore from reme.components import ApplicationContext from reme.components.runtime_context import RuntimeContext from reme.enumeration import LinkScopeEnum -from reme.schema import FileChunk, FileLink, FileNode +from reme.schema import FileChunk, FileLink, FileNode, Response from reme.steps.index import ( AddDraftStep, Bm25SearchStep, @@ -984,3 +986,276 @@ def test_search_step_non_strict_date_filter_not_in_search_filter(): assert "strict_date_filter" not in sf asyncio.run(run()) + + +def test_search_v2_step_compresses_session_bodies_when_injected_flag_set(): + """With ``_search._compress.session`` truthy, session bodies are replaced by the + compressor job output while headers and non-session entries stay untouched.""" + + async def run(): + session = _chunk("s1", "session/dialog/s1.jsonl", "hello world dialog text", "vector", 0.9, line=2) + note = _chunk("n1", "daily/a.md", "note text", "vector", 0.8) + store = FakeSearchStore(vector_results=[session, note]) + step = SearchV2Step(file_store=store, expand_links=False) + calls: list[dict] = [] + + async def fake_run_job(name, /, **kwargs): + calls.append({"name": name, **kwargs}) + return Response(success=True, answer="compressed!") + + step.run_job = fake_run_job + ctx = RuntimeContext( + query="alpha", + limit=5, + _search={"_compress": {"session": "true"}, "queries": ["orig question"]}, + ) + + resp = await step(ctx) + + assert resp.success is True + assert "compressed!" in resp.answer + assert "hello world dialog text" not in resp.answer + assert "note text" in resp.answer + assert "session/dialog/s1.jsonl:2-2" in resp.answer + assert calls == [ + {"name": "compressor", "text": "hello world dialog text", "queries": ["orig question", "alpha"]}, + ] + + asyncio.run(run()) + + +def test_search_v2_step_query_independent_compression_passes_no_queries(): + """With ``_search.type`` = ``query-independent`` the compressor receives no queries; + any other (or missing) type keeps the query-aware behavior.""" + + async def run(): + session = _chunk("s1", "session/dialog/s1.jsonl", "hello world dialog text", "vector", 0.9) + store = FakeSearchStore(vector_results=[session]) + step = SearchV2Step(file_store=store, expand_links=False) + calls: list[dict] = [] + + async def fake_run_job(name, /, **kwargs): + calls.append({"name": name, **kwargs}) + return Response(success=True, answer="compressed!") + + step.run_job = fake_run_job + ctx = RuntimeContext( + query="alpha", + limit=5, + _search={"_compress": {"session": "true"}, "queries": ["orig question"], "type": "query-independent"}, + ) + + resp = await step(ctx) + + assert "compressed!" in resp.answer + assert calls == [{"name": "compressor", "text": "hello world dialog text", "queries": []}] + + # Explicit query-aware type behaves exactly like the default. + calls.clear() + ctx = RuntimeContext( + query="alpha", + limit=5, + _search={"_compress": {"session": "true"}, "queries": ["orig question"], "type": "query-aware"}, + ) + + await step(ctx) + + assert calls == [ + {"name": "compressor", "text": "hello world dialog text", "queries": ["orig question", "alpha"]}, + ] + + asyncio.run(run()) + + +def test_search_v2_step_keeps_original_body_when_compressed_is_longer(): + """A compressed body longer than the original is rejected in favor of the original.""" + + async def run(): + session = _chunk("s1", "session/dialog/s1.jsonl", "short", "vector", 0.9) + store = FakeSearchStore(vector_results=[session]) + step = SearchV2Step(file_store=store, expand_links=False) + + async def fake_run_job(name, /, **kwargs): # pylint: disable=unused-argument + return Response(success=True, answer="a much longer compressed output") + + step.run_job = fake_run_job + ctx = RuntimeContext(query="alpha", _search={"_compress": {"session": True}}) + + resp = await step(ctx) + + assert "short" in resp.answer + assert "much longer" not in resp.answer + + asyncio.run(run()) + + +def test_search_v2_step_compression_strips_input_and_adds_marker(): + """The compressor receives the stripped rendered body; adopted output is + prefixed with the ``compressed session chunk:`` marker only (no line numbers).""" + + async def run(): + session = FileChunk( + id="s1", + path="session/dialog/s1.jsonl", + text="first message line\nsecond message line", + start_line=4, + end_line=5, + scores={"vector": 0.9, "score": 0.9}, + ) + store = FakeSearchStore(vector_results=[session]) + step = SearchV2Step(file_store=store, expand_links=False) + calls: list[dict] = [] + + async def fake_run_job(name, /, **kwargs): + calls.append({"name": name, **kwargs}) + return Response(success=True, answer="first compact\nsecond compact\n") + + step.run_job = fake_run_job + ctx = RuntimeContext(query="alpha", _search={"_compress": {"session": "true"}}) + + resp = await step(ctx) + + assert calls[0]["text"] == "first message line\nsecond message line" + assert "compressed session chunk:\nfirst compact\nsecond compact" in resp.answer + + asyncio.run(run()) + + +def test_search_v2_step_compression_input_is_line_aligned_with_session_jsonl(): + """Serialized Msg jsonl lines render one message per line (newlines flattened), + so compressor input line ``i`` maps to session file line ``start_line + i``.""" + + async def run(): + m1 = Msg(name="user", role="user", content=[{"type": "text", "text": "line one\nline two"}]) + m2 = Msg(name="assistant", role="assistant", content=[{"type": "text", "text": "long answer"}]) + session = FileChunk( + id="s1", + path="session/dialog/s1.jsonl", + text=m1.model_dump_json() + "\n" + m2.model_dump_json(), + start_line=7, + end_line=8, + scores={"vector": 0.9, "score": 0.9}, + ) + store = FakeSearchStore(vector_results=[session]) + step = SearchV2Step(file_store=store, expand_links=False) + calls: list[dict] = [] + + async def fake_run_job(name, /, **kwargs): + calls.append({"name": name, **kwargs}) + return Response(success=True, answer="[user @ t] u\n[assistant @ t] a") + + step.run_job = fake_run_job + ctx = RuntimeContext(query="alpha", _search={"_compress": {"session": "true"}}) + + resp = await step(ctx) + + sent_lines = calls[0]["text"].splitlines() + assert len(sent_lines) == 2 + assert sent_lines[0].startswith("[user @") and "line one line two" in sent_lines[0] + assert sent_lines[1].startswith("[assistant @") and "long answer" in sent_lines[1] + assert "compressed session chunk:\n[user @ t] u\n[assistant @ t] a" in resp.answer + + asyncio.run(run()) + + +def test_search_v2_step_adopts_shorter_output_regardless_of_line_count(): + """A shorter compressor output is adopted even when its line count differs.""" + + async def run(): + session = FileChunk( + id="s1", + path="session/dialog/s1.jsonl", + text="first message line\nsecond message line", + start_line=4, + end_line=5, + scores={"vector": 0.9, "score": 0.9}, + ) + store = FakeSearchStore(vector_results=[session]) + step = SearchV2Step(file_store=store, expand_links=False) + + async def fake_run_job(name, /, **kwargs): # pylint: disable=unused-argument + return Response(success=True, answer="merged into one line") + + step.run_job = fake_run_job + ctx = RuntimeContext(query="alpha", _search={"_compress": {"session": "true"}}) + + resp = await step(ctx) + + assert "compressed session chunk:\nmerged into one line" in resp.answer + assert "first message line" not in resp.answer + + asyncio.run(run()) + + +def test_search_v2_step_skips_compression_without_injected_flag(): + """Without the injected ``_search`` payload the compressor job is never invoked.""" + + async def run(): + session = _chunk("s1", "session/dialog/s1.jsonl", "dialog text", "vector", 0.9) + store = FakeSearchStore(vector_results=[session]) + step = SearchV2Step(file_store=store, expand_links=False) + + async def fake_run_job(name, /, **kwargs): # pylint: disable=unused-argument + raise AssertionError("compressor job must not be called") + + step.run_job = fake_run_job + ctx = RuntimeContext(query="alpha") + + resp = await step(ctx) + + assert "dialog text" in resp.answer + + asyncio.run(run()) + + +def test_search_v2_step_keeps_original_body_when_compression_fails(): + """A failed compressor response (success=False) never replaces the body.""" + + async def run(): + session = _chunk("s1", "session/dialog/s1.jsonl", "dialog text", "vector", 0.9) + store = FakeSearchStore(vector_results=[session]) + step = SearchV2Step(file_store=store, expand_links=False) + + async def fake_run_job(name, /, **kwargs): # pylint: disable=unused-argument + return Response(success=False, answer="boom") + + step.run_job = fake_run_job + ctx = RuntimeContext(query="alpha", _search={"_compress": {"session": "true"}}) + + resp = await step(ctx) + + assert "dialog text" in resp.answer + assert "boom" not in resp.answer + + asyncio.run(run()) + + +def test_search_v2_step_keeps_original_body_when_compression_raises(): + """A compressor job that raises (e.g. temporary LLM outage) keeps the original + body for that entry while other entries are still compressed; the search itself + never fails.""" + + async def run(): + failing = _chunk("s1", "session/dialog/s1.jsonl", "dialog text one", "vector", 0.9, line=2) + ok = _chunk("s2", "session/dialog/s2.jsonl", "dialog text two", "vector", 0.8, line=1) + store = FakeSearchStore(vector_results=[failing, ok]) + step = SearchV2Step(file_store=store, expand_links=False) + + async def fake_run_job(_name, /, **kwargs): + if "one" in kwargs.get("text", ""): + raise RuntimeError("temporary LLM outage") + return Response(success=True, answer="compressed!") + + step.run_job = fake_run_job + ctx = RuntimeContext(query="alpha", _search={"_compress": {"session": "true"}}) + + resp = await step(ctx) + + assert resp.success is True + # The failing entry keeps its original body. + assert "dialog text one" in resp.answer + # The healthy entry is still compressed. + assert "compressed!" in resp.answer + assert "dialog text two" not in resp.answer + + asyncio.run(run()) diff --git a/tests/unit/test_source_format_merge.py b/tests/unit/test_source_format_merge.py index f259eaad..98b7eaca 100644 --- a/tests/unit/test_source_format_merge.py +++ b/tests/unit/test_source_format_merge.py @@ -1,60 +1,68 @@ -"""Unit tests for session-chunk merging in ``format_chunks_answer``. +"""Unit tests for session-chunk merging via ``merge_session_chunk_intervals``. Session chunks (``*.jsonl`` under the dialog dir) whose line ranges overlap, -contain one another, or are adjacent are merged into their union before -rendering. Bodies here are plain (non-``Msg``) text, so ``render_chunk_body`` -falls back to the raw chunk text — letting these tests assert the union -content and line order directly. +contain one another, or are adjacent are merged into their union by the caller +(``merge_session_chunk_intervals``) before rendering with +``render_chunk_entries`` + ``join_chunk_entries``. Bodies here are plain +(non-``Msg``) text lines, which pass through stripped — letting these tests +assert the union content and line order directly. """ from reme.schema import FileChunk -from reme.steps.index._source_format import format_chunks_answer +from reme.steps.index._source_format import join_chunk_entries, merge_session_chunk_intervals, render_chunk_entries _DIALOG_DIR = "session" +def _render(chunks: list[FileChunk], dialog_dir: str, **kwargs) -> str: + """Merge session chunks, render entries, then join — mirroring the search steps.""" + return join_chunk_entries( + render_chunk_entries(merge_session_chunk_intervals(chunks, dialog_dir), dialog_dir, **kwargs), + ) + + def _chunk(start: int, end: int, text: str, score: float = 1.0, path: str = "session/s1.jsonl") -> FileChunk: return FileChunk(path=path, start_line=start, end_line=end, text=text, scores={"score": score}) def test_overlapping_session_chunks_merge_into_union_without_duplicates(): """Overlapping ranges collapse to one passage; the shared line is shown once, in order.""" - a = _chunk(1, 3, "L1\nL2\nL3\n") - b = _chunk(3, 5, "L3\nL4\nL5\n") + a = _chunk(1, 3, "m1\nm2\nm3\n") + b = _chunk(3, 5, "m3\nm4\nm5\n") - answer = format_chunks_answer([a, b], _DIALOG_DIR, include_source=False) + answer = _render([a, b], _DIALOG_DIR, include_source=False) - assert answer == "L1\nL2\nL3\nL4\nL5\n" + assert answer == "m1\nm2\nm3\nm4\nm5" def test_contained_session_chunk_is_absorbed_by_the_larger_range(): """When one range fully contains another, only the union (the larger) is shown.""" - big = _chunk(1, 5, "L1\nL2\nL3\nL4\nL5\n") - small = _chunk(2, 4, "L2\nL3\nL4\n") + big = _chunk(1, 5, "m1\nm2\nm3\nm4\nm5\n") + small = _chunk(2, 4, "m2\nm3\nm4\n") - answer = format_chunks_answer([big, small], _DIALOG_DIR, include_source=False) + answer = _render([big, small], _DIALOG_DIR, include_source=False) - assert answer == "L1\nL2\nL3\nL4\nL5\n" + assert answer == "m1\nm2\nm3\nm4\nm5" def test_adjacent_session_chunks_merge_end_plus_one_equals_next_start(): """Gap-free consecutive chunks (prev.end + 1 == next.start) merge into one union.""" - a = _chunk(1, 3, "L1\nL2\nL3\n") - b = _chunk(4, 6, "L4\nL5\nL6\n") + a = _chunk(1, 3, "m1\nm2\nm3\n") + b = _chunk(4, 6, "m4\nm5\nm6\n") - answer = format_chunks_answer([a, b], _DIALOG_DIR, include_source=False) + answer = _render([a, b], _DIALOG_DIR, include_source=False) - assert answer == "L1\nL2\nL3\nL4\nL5\nL6\n" + assert answer == "m1\nm2\nm3\nm4\nm5\nm6" def test_session_chunks_with_a_gap_are_not_merged(): """A missing line between ranges (start > end + 1) keeps the chunks separate.""" - a = _chunk(1, 3, "L1\nL2\nL3\n") - b = _chunk(5, 6, "L5\nL6\n") # line 4 missing -> not adjacent + a = _chunk(1, 3, "m1\nm2\nm3\n") + b = _chunk(5, 6, "m5\nm6\n") # line 4 missing -> not adjacent - answer = format_chunks_answer([a, b], _DIALOG_DIR, include_source=False) + answer = _render([a, b], _DIALOG_DIR, include_source=False) - assert answer == "L1\nL2\nL3\n\n\nL5\nL6\n" + assert answer == "m1\nm2\nm3\n\nm5\nm6" def test_session_chunks_from_different_files_are_not_merged(): @@ -62,9 +70,9 @@ def test_session_chunks_from_different_files_are_not_merged(): a = _chunk(1, 3, "A1\nA2\nA3\n", path="session/s1.jsonl") b = _chunk(2, 4, "B2\nB3\nB4\n", path="session/s2.jsonl") - answer = format_chunks_answer([a, b], _DIALOG_DIR, include_source=False) + answer = _render([a, b], _DIALOG_DIR, include_source=False) - assert answer == "A1\nA2\nA3\n\n\nB2\nB3\nB4\n" + assert answer == "A1\nA2\nA3\n\nB2\nB3\nB4" def test_non_session_chunks_are_never_merged(): @@ -72,28 +80,28 @@ def test_non_session_chunks_are_never_merged(): a = _chunk(1, 3, "M1\nM2\nM3\n", path="daily/a.md") b = _chunk(2, 4, "M2\nM3\nM4\n", path="daily/a.md") - answer = format_chunks_answer([a, b], _DIALOG_DIR, include_source=False) + answer = _render([a, b], _DIALOG_DIR, include_source=False) - assert answer == "M1\nM2\nM3\n\n\nM2\nM3\nM4\n" + assert answer == "M1\nM2\nM3\n\nM2\nM3\nM4" def test_merge_preserves_line_order_regardless_of_input_rank_order(): """A later, higher-ranked chunk does not reorder union content; lines stay chronological.""" - later = _chunk(3, 5, "L3\nL4\nL5\n", score=9.0) - earlier = _chunk(1, 3, "L1\nL2\nL3\n", score=1.0) + later = _chunk(3, 5, "m3\nm4\nm5\n", score=9.0) + earlier = _chunk(1, 3, "m1\nm2\nm3\n", score=1.0) # Higher-scored later-range chunk is listed first (as a ranker would). - answer = format_chunks_answer([later, earlier], _DIALOG_DIR, include_source=False) + answer = _render([later, earlier], _DIALOG_DIR, include_source=False) - assert answer == "L1\nL2\nL3\nL4\nL5\n" + assert answer == "m1\nm2\nm3\nm4\nm5" def test_merged_header_spans_the_union_range_and_keeps_best_score(): """With source headers, the merged entry reports the union range and the top score.""" - a = _chunk(1, 3, "L1\nL2\nL3\n", score=2.0) - b = _chunk(3, 5, "L3\nL4\nL5\n", score=7.0) + a = _chunk(1, 3, "m1\nm2\nm3\n", score=2.0) + b = _chunk(3, 5, "m3\nm4\nm5\n", score=7.0) - answer = format_chunks_answer([a, b], _DIALOG_DIR, include_source=True) + answer = _render([a, b], _DIALOG_DIR, include_source=True) assert answer.count("==========") == 2 # exactly one header (open + close markers) assert "session/s1.jsonl:1-5" in answer @@ -102,13 +110,13 @@ def test_merged_header_spans_the_union_range_and_keeps_best_score(): def test_separate_intervals_in_same_file_stay_separate(): """Two disjoint interval clusters in one file yield two merged units, ordered by line.""" - a = _chunk(1, 2, "L1\nL2\n") - b = _chunk(3, 4, "L3\nL4\n") # adjacent to a -> merges with a into 1-4 - c = _chunk(10, 11, "L10\nL11\n") # far away -> separate + a = _chunk(1, 2, "m1\nm2\n") + b = _chunk(3, 4, "m3\nm4\n") # adjacent to a -> merges with a into 1-4 + c = _chunk(10, 11, "m10\nm11\n") # far away -> separate - answer = format_chunks_answer([a, b, c], _DIALOG_DIR, include_source=False) + answer = _render([a, b, c], _DIALOG_DIR, include_source=False) - assert answer == "L1\nL2\nL3\nL4\n\n\nL10\nL11\n" + assert answer == "m1\nm2\nm3\nm4\n\nm10\nm11" def test_same_file_units_stay_adjacent_and_sorted_even_when_interleaved_by_rank(): @@ -120,11 +128,11 @@ def test_same_file_units_stay_adjacent_and_sorted_even_when_interleaved_by_rank( s1_low = _chunk(1, 3, "S1a\nS1b\nS1c\n", score=1.0, path="session/s1.jsonl") # Rank order (as a ranker would emit, by score desc): s1[10-12], s2[1-3], s1[1-3]. - answer = format_chunks_answer([s1_high, s2_mid, s1_low], _DIALOG_DIR, include_source=False) + answer = _render([s1_high, s2_mid, s1_low], _DIALOG_DIR, include_source=False) # s1's two units are adjacent and sorted by start_line (1-3 before 10-12), # placed at s1's earliest rank (0), so the whole s1 block precedes s2. - assert answer == "S1a\nS1b\nS1c\n\n\nS1x\nS1y\nS1z\n\n\nS2a\nS2b\nS2c\n" + assert answer == "S1a\nS1b\nS1c\n\nS1x\nS1y\nS1z\n\nS2a\nS2b\nS2c" def test_same_file_units_adjacency_with_source_headers(): @@ -133,7 +141,7 @@ def test_same_file_units_adjacency_with_source_headers(): s2_mid = _chunk(1, 3, "S2a\nS2b\nS2c\n", score=5.0, path="session/s2.jsonl") s1_low = _chunk(1, 3, "S1a\nS1b\nS1c\n", score=1.0, path="session/s1.jsonl") - answer = format_chunks_answer([s1_high, s2_mid, s1_low], _DIALOG_DIR, include_source=True) + answer = _render([s1_high, s2_mid, s1_low], _DIALOG_DIR, include_source=True) headers = [line for line in answer.splitlines() if line.startswith("==========")] assert headers[0].split(" [")[0] == "========== session/s1.jsonl:1-3" diff --git a/tests/unit/test_workspace_web_steps.py b/tests/unit/test_workspace_web_steps.py new file mode 100644 index 00000000..7071ce8e --- /dev/null +++ b/tests/unit/test_workspace_web_steps.py @@ -0,0 +1,133 @@ +"""Focused contracts used by the local ReMe workspace web client.""" + +import asyncio +import os +from datetime import datetime + +from reme.components.agent_wrapper import BaseAgentWrapper +from reme.components.application_context import ApplicationContext +from reme.components.runtime_context import RuntimeContext +from reme.enumeration import ChunkEnum +from reme.schema import StreamChunk +from reme.steps.common.chat import ChatStep +from reme.steps.file_io.load import LoadStep +from reme.steps.file_io.save import SaveStep + + +class _StreamingAgent(BaseAgentWrapper): + """Minimal agent that records resume/tool arguments and emits rich chunks.""" + + def __init__(self): + super().__init__() + self.reply_kwargs = {} + + async def reply(self, inputs, **kwargs) -> dict: + raise AssertionError("ChatStep must always use reply_stream()") + + async def reply_stream(self, inputs, **kwargs): + self.reply_kwargs = kwargs + yield StreamChunk(chunk_type=ChunkEnum.TOOL_CALL, chunk="{}", tool_call_id="tool-1", tool_call_name="search") + yield StreamChunk(chunk_type=ChunkEnum.CONTENT, chunk="hello", session_id="session-new") + + +def test_save_step_preserves_complete_markdown(tmp_path): + """The editor save endpoint writes frontmatter and body verbatim.""" + + async def run(): + target = tmp_path / "daily" / "note.md" + target.parent.mkdir(parents=True) + target.write_text("old", encoding="utf-8") + expected = datetime.fromtimestamp(target.stat().st_mtime).isoformat() + content = "---\nname: note\ncustom: keep\n---\n\n# Updated\n" + + response = await SaveStep(app_context=ApplicationContext(workspace_dir=str(tmp_path)))( + path="daily/note.md", + content=content, + expected_mtime=expected, + ) + + assert response.success is True + assert target.read_text(encoding="utf-8") == content + assert response.metadata["path"] == "daily/note.md" + assert response.metadata["mtime"] + + asyncio.run(run()) + + +def test_load_step_returns_complete_content_and_mtime(tmp_path): + """Editor loads are complete and include the concurrency token for save.""" + + async def run(): + content = "# Large enough\n" + ("memory line\n" * 5000) + target = tmp_path / "note.md" + target.write_text(content, encoding="utf-8") + + response = await LoadStep(app_context=ApplicationContext(workspace_dir=str(tmp_path)))(path="note.md") + + assert response.success is True + assert response.answer == content + assert response.metadata["size"] == target.stat().st_size + assert response.metadata["mtime"] + + asyncio.run(run()) + + +def test_load_step_rejects_file_larger_than_editor_limit(tmp_path): + """Oversized files fail explicitly instead of returning editable truncation.""" + + async def run(): + (tmp_path / "note.md").write_text("0123456789", encoding="utf-8") + + response = await LoadStep(app_context=ApplicationContext(workspace_dir=str(tmp_path)))( + path="note.md", + max_bytes=5, + ) + + assert response.success is False + assert response.metadata["code"] == "file_too_large" + + asyncio.run(run()) + + +def test_save_step_rejects_external_change(tmp_path): + """An obsolete mtime cannot overwrite a file changed by another editor.""" + + async def run(): + target = tmp_path / "note.md" + target.write_text("first", encoding="utf-8") + expected = datetime.fromtimestamp(target.stat().st_mtime).isoformat() + target.write_text("external", encoding="utf-8") + newer = target.stat().st_mtime + 2 + os.utime(target, (newer, newer)) + + response = await SaveStep(app_context=ApplicationContext(workspace_dir=str(tmp_path)))( + path="note.md", + content="browser edit", + expected_mtime=expected, + ) + + assert response.success is False + assert response.metadata["code"] == "file_conflict" + assert target.read_text(encoding="utf-8") == "external" + + asyncio.run(run()) + + +def test_chat_step_streams_rich_chunks_and_resumes_session(): + """Web chat keeps StreamChunk metadata and exposes only read-only tools.""" + + async def run(): + agent = _StreamingAgent() + queue = asyncio.Queue() + context = RuntimeContext(stream_queue=queue, query="hello", session_id="session-old") + + response = await ChatStep(agent_wrapper=agent)(context) + + chunks = [await queue.get(), await queue.get()] + assert response.answer == "hello" + assert chunks[0].tool_call_name == "search" + assert chunks[1].session_id == "session-new" + assert agent.reply_kwargs["resume"] == "session-old" + assert agent.reply_kwargs["job_tools"] == ["search", "list", "read", "stat", "traverse"] + + asyncio.run(run())