diff --git a/.gitignore b/.gitignore index badafa78..d66778a8 100644 --- a/.gitignore +++ b/.gitignore @@ -2,8 +2,8 @@ .DS_Store .idea/ .vscode/ -*.code-workspace .qoder/ +*.code-workspace # Local environment .env @@ -54,5 +54,17 @@ vault/ docs/_build/ site/ -evaluation/ -datasets/ +# Memory workspaces (keep dirs, ignore contents) +benchmark/memory_workspaces/ + +# LongMemEval raw data (download via download.py) +benchmark/datasets/longmemeval/longmemeval_s_reme_cleaned.json + +# BEAM dataset (clone via git) +benchmark/datasets/BEAM/ + +# Benchmark outputs +benchmark/results/ + +# integration tests outputs +tests/integration/logs/ diff --git a/benchmark/README.md b/benchmark/README.md new file mode 100644 index 00000000..bbbc101a --- /dev/null +++ b/benchmark/README.md @@ -0,0 +1,145 @@ +[中文版 / Chinese version](./README_ZH.md) + +# ReMe Benchmarks + +Reproduction guide for the two memory benchmarks shipped with ReMe: + +- **LongMemEval** — long-term memory over multi-session chat histories. +- **BEAM** — memory capability over long-context chat cases with rubric-based judging. + +Each benchmark runs its own end-to-end pipeline: ingest sessions into an isolated +per-item workspace, answer probing questions via an agentic (ReAct) mode, +then score answers with an LLM-as-judge. + +## 1. Prerequisites + +Install ReMe with dev + core extras (Python 3.11+): + +```bash +pip install -e ".[dev,core]" +``` + +Configure model credentials in a project-root `.env` file (copied from `example.env`). +The runners auto-load `.env` from the repository root. Required variables typically include: + +```bash +LLM_API_KEY=... +LLM_BASE_URL=... +EMBEDDING_API_KEY=... +EMBEDDING_BASE_URL=... +``` + +Model names and component wiring live in the ReMe configs referenced by each benchmark +(`reme/config/lme.yaml` and `reme/config/beam.yaml`). + +## 2. Download Datasets + +See [`datasets/README_EN.md`](datasets/README_EN.md) for full details. + +**LongMemEval** (downloaded from a HuggingFace mirror): + +```bash +cd benchmark/datasets/longmemeval +python download.py # downloads the cleaned-S dataset; skips if already present +``` + +**BEAM** (public repository, cloned into `benchmark/datasets/`): + +```bash +cd benchmark/datasets +git clone https://github.com/mohammadtavakoli78/BEAM.git +``` + +## 3. Run LongMemEval + +From the repository root: + +```bash +python benchmark/longmemeval/run.py +python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml +python benchmark/longmemeval/run.py -q # quiet: only eval-level logs +python benchmark/longmemeval/run.py --log-level WARNING # reduce eval runner logs +python benchmark/longmemeval/run.py --reme-log-level WARNING # reduce reme internal logs +python benchmark/longmemeval/run.py --eval_only # reuse existing workspaces, query + judge only +``` + +### Pipeline + +1. Load the dataset (ground truth is embedded in the data file). +2. For each item, create an isolated workspace and ingest sessions in chronological order. +3. Trigger `auto_dream` when consecutive sessions cross the configured hour (default 23:00). +4. Answer each question via agentic (ReAct) mode. +5. Judge the answer (binary yes/no) with the `answer_judge` job and print per-type accuracy. + +### Key config — `benchmark/longmemeval/config.yaml` + +| Key | Meaning | +| --- | --- | +| `dataset.path` | Dataset file to evaluate (e.g. `longmemeval_s_reme_cleaned.json`); ground truth is included. | +| `dataset.start_index` / `num_items` | Slice of items to evaluate. | +| `dataset.question_types` | Filter by question type; empty = all. | +| `dataset.workspace_root` | Per-item workspace root (`benchmark/memory_workspaces/longmemeval-s`). | +| `evaluation.num_workers` | `0` = auto (cpu-2), `1` = sequential, `>1` = parallel. | +| `evaluation.filter_future_sessions` | Only ingest sessions with timestamp ≤ `question_date`. | +| `reme.config` | ReMe config used (`lme.yaml`). | +| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | Dream triggering behavior. | +| `output.dir` | Results directory (`benchmark/results/longmemeval`). | + +## 4. Run BEAM + +From the repository root: + +```bash +python benchmark/beam/run.py +python benchmark/beam/run.py --config benchmark/beam/config.yaml +python benchmark/beam/run.py -q # quiet +python benchmark/beam/run.py --eval_only # reuse existing workspaces, query + judge only +``` + +### Pipeline + +1. For each case, load `chat.json` and convert each batch into a ReMe session. +2. Ingest sessions in chronological order into an isolated workspace, then `digest_update`. +3. Answer each probing question via agentic (ReAct) mode. +4. Score answers with BEAM's rubric-based `answer_judge` job and print per-type averages. + +### Key config — `benchmark/beam/config.yaml` + +| Key | Meaning | +| --- | --- | +| `dataset.beam_root` | BEAM dataset root (`benchmark/datasets/BEAM`). | +| `dataset.chat_size` | Variant to run: `100K` / `500K` / `1M` / `10M`. | +| `dataset.case_ids` | Specific cases (e.g. `["1","2"]`); empty = all cases. | +| `dataset.start_index` / `num_items` | Case pagination (`num_items` `0` = all). | +| `dataset.workspace_root` | Per-case workspace root (`benchmark/memory_workspaces/beam`). | +| `evaluation.num_workers` | `0` = auto, `1` = sequential, `>1` = parallel. | +| `reme.config` | ReMe config used (`beam.yaml`). | +| `output.dir` | Results directory (`benchmark/results/beam`). | + +## 5. Outputs & Logs + +- **Results**: JSON files written to `output.dir` + (`results_.json` for LongMemEval, + `results__.json` for BEAM). A summary with per-type + accuracy/score is also printed to the console. +- **Logs**: when `output.log_to_file` is enabled, per-run logs are written to + `logs/_/` (a `runner.log` plus one `worker-.log` + per worker process). + +## 6. Stopping a Run + +Parallel runs spawn a process tree. To terminate a run and all its workers cleanly: + +```bash +bash benchmark/kill.sh +``` + +The script gracefully sends `SIGTERM` to the whole process tree, then escalates to +`SIGKILL` for any process that does not exit within 5 seconds. + +## 7. Reference Results + +Recorded evaluation results are available in: + +- [`result-longmemeval.md`](./result-longmemeval.md) +- [`result-beam.md`](./result-beam.md) diff --git a/benchmark/README_ZH.md b/benchmark/README_ZH.md new file mode 100644 index 00000000..c384a01c --- /dev/null +++ b/benchmark/README_ZH.md @@ -0,0 +1,142 @@ +# ReMe 评测复现说明 + +ReMe 内置两个记忆能力评测基准的复现指南： + +- **LongMemEval** —— 面向多轮多会话历史的长期记忆能力评测。 +- **BEAM** —— 面向长上下文对话场景、基于评分细则（rubric）打分的记忆能力评测。 + +每个基准都会运行完整的端到端流程：将会话摄入独立的按条目隔离的工作区， +以 agentic（ReAct）模式回答探测问题， +最后由 LLM-as-judge 对答案进行打分。 + +## 1. 环境准备 + +安装 ReMe 及 dev + core 附加依赖（Python 3.11+）： + +```bash +pip install -e ".[dev,core]" +``` + +在项目根目录配置 `.env`（可从 `example.env` 复制），运行脚本会自动从仓库根目录加载 `.env`。 +通常需要以下变量： + +```bash +LLM_API_KEY=... +LLM_BASE_URL=... +EMBEDDING_API_KEY=... +EMBEDDING_BASE_URL=... +``` + +模型名称与组件装配位于各基准引用的 ReMe 配置中 +（`reme/config/lme.yaml` 与 `reme/config/beam.yaml`）。 + +## 2. 下载数据集 + +完整说明见 [`datasets/README.md`](datasets/README.md)。 + +**LongMemEval**（从 HuggingFace 镜像下载）： + +```bash +cd benchmark/datasets/longmemeval +python download.py # 下载 cleaned-S 数据文件，已存在则自动跳过 +``` + +**BEAM**（公开仓库，clone 到 `benchmark/datasets/` 下）： + +```bash +cd benchmark/datasets +git clone https://github.com/mohammadtavakoli78/BEAM.git +``` + +## 3. 运行 LongMemEval + +在仓库根目录执行： + +```bash +python benchmark/longmemeval/run.py +python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml +python benchmark/longmemeval/run.py -q # 安静模式：仅评测级日志 +python benchmark/longmemeval/run.py --log-level WARNING # 降低评测 runner 日志 +python benchmark/longmemeval/run.py --reme-log-level WARNING # 降低 reme 内部日志 +python benchmark/longmemeval/run.py --eval_only # 复用已有工作区，仅执行查询 + 评判 +``` + +### 流程 + +1. 加载数据集（ground truth 已内嵌在数据文件中）。 +2. 为每个条目创建独立工作区，按时间顺序摄入会话。 +3. 当相邻会话跨越配置的时刻（默认 23:00）时触发 `auto_dream`。 +4. 以 agentic（ReAct）模式回答每个问题。 +5. 通过 `answer_judge` 任务对答案做二元（yes/no）评判，并输出各类型准确率。 + +### 关键配置 —— `benchmark/longmemeval/config.yaml` + +| 配置项 | 含义 | +| --- | --- | +| `dataset.path` | 待评测的数据集文件（如 `longmemeval_s_reme_cleaned.json`），已包含 ground truth。 | +| `dataset.start_index` / `num_items` | 评测条目的切片范围。 | +| `dataset.question_types` | 按问题类型过滤，空表示全部。 | +| `dataset.workspace_root` | 条目工作区根目录（`benchmark/memory_workspaces/longmemeval-s`）。 | +| `evaluation.num_workers` | `0` = 自动（cpu-2），`1` = 串行，`>1` = 并行。 | +| `evaluation.filter_future_sessions` | 仅摄入时间戳 ≤ `question_date` 的会话。 | +| `reme.config` | 使用的 ReMe 配置（`lme.yaml`）。 | +| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | dream 触发行为。 | +| `output.dir` | 结果目录（`benchmark/results/longmemeval`）。 | + +## 4. 运行 BEAM + +在仓库根目录执行： + +```bash +python benchmark/beam/run.py +python benchmark/beam/run.py --config benchmark/beam/config.yaml +python benchmark/beam/run.py -q # 安静模式 +python benchmark/beam/run.py --eval_only # 复用已有工作区，仅执行查询 + 评判 +``` + +### 流程 + +1. 为每个 case 加载 `chat.json`，将每个 batch 转换为一个 ReMe 会话。 +2. 按时间顺序将会话摄入独立工作区，随后执行 `digest_update`。 +3. 以 agentic（ReAct）模式回答每个探测问题。 +4. 通过 BEAM 基于 rubric 的 `answer_judge` 任务打分，并输出各类型平均分。 + +### 关键配置 —— `benchmark/beam/config.yaml` + +| 配置项 | 含义 | +| --- | --- | +| `dataset.beam_root` | BEAM 数据集根目录（`benchmark/datasets/BEAM`）。 | +| `dataset.chat_size` | 运行的变体：`100K` / `500K` / `1M` / `10M`。 | +| `dataset.case_ids` | 指定 case（如 `["1","2"]`），空表示全部。 | +| `dataset.start_index` / `num_items` | case 分页（`num_items` 为 `0` 表示全部）。 | +| `dataset.workspace_root` | case 工作区根目录（`benchmark/memory_workspaces/beam`）。 | +| `evaluation.num_workers` | `0` = 自动，`1` = 串行，`>1` = 并行。 | +| `reme.config` | 使用的 ReMe 配置（`beam.yaml`）。 | +| `output.dir` | 结果目录（`benchmark/results/beam`）。 | + +## 5. 输出与日志 + +- **结果**：JSON 文件写入 `output.dir` + （LongMemEval 为 `results_.json`， + BEAM 为 `results__.json`）。同时控制台会打印含各类型 + 准确率/分数的汇总。 +- **日志**：当 `output.log_to_file` 开启时，每次运行的日志写入 + `logs/_/`（包含一个 `runner.log` 及每个 worker 进程的 + `worker-.log`）。 + +## 6. 终止运行 + +并行运行会派生进程树。若要干净地终止某次运行及其全部 worker： + +```bash +bash benchmark/kill.sh +``` + +该脚本会先向整个进程树发送 `SIGTERM` 优雅终止，对 5 秒内未退出的进程再升级为 `SIGKILL`。 + +## 7. 参考结果 + +已记录的评测结果见： + +- [`result-longmemeval.md`](./result-longmemeval.md) +- [`result-beam.md`](./result-beam.md) diff --git a/benchmark/beam/config.yaml b/benchmark/beam/config.yaml new file mode 100644 index 00000000..32e7620b --- /dev/null +++ b/benchmark/beam/config.yaml @@ -0,0 +1,23 @@ +# BEAM evaluation configuration +# This file controls what/how to evaluate. + +dataset: + beam_root: "benchmark/datasets/BEAM" # BEAM dataset root + chat_size: "1M" # 100K | 500K | 1M | 10M (dataset variant) + case_ids: [] # empty = all cases; or ["1", "2", "3"] + start_index: 0 # first case index (for pagination) + num_items: 0 # 0 = all cases; >0 = limit + workspace_root: "benchmark/memory_workspaces/beam" # workspace root for case workspaces + +evaluation: + num_workers: 18 # 0 = auto; 1 = sequential; >1 = parallel (per-case) + +reme: + config: "beam.yaml" # reme config (in reme/config/) + +output: + dir: "benchmark/results/beam" + log_dir: "logs" # log directory (relative to project root) + log_prefix: "beam" # benchmark name used in log filenames + log_to_console: true + log_to_file: true diff --git a/benchmark/beam/run.py b/benchmark/beam/run.py new file mode 100644 index 00000000..45c74f10 --- /dev/null +++ b/benchmark/beam/run.py @@ -0,0 +1,791 @@ +"""BEAM evaluation runner for ReMe. + +Evaluates ReMe's memory capability using the BEAM dataset. +Each case gets an isolated workspace; chat.json batches are ingested as +sessions in chronological order; finally probing questions are answered +via an agentic (ReAct) approach, then +judged by BEAM's rubric-based LLM-as-judge. + +Usage: + python benchmark/beam/run.py + python benchmark/beam/run.py --config benchmark/beam/config.yaml + python benchmark/beam/run.py -q # quiet: only eval-level logs + python benchmark/beam/run.py --log-level WARNING # reduce eval runner logs + python benchmark/beam/run.py --reme-log-level WARNING # reduce reme internal logs + python benchmark/beam/run.py --eval_only # query+judge only, reuse existing workspace +""" + +import json +import logging +import os +import re +import shutil +import time +import threading +from datetime import datetime +from pathlib import Path + +import yaml +from dotenv import load_dotenv + +# Load .env from project root +_PROJECT_ROOT = Path(__file__).parent.parent.parent +load_dotenv(_PROJECT_ROOT / ".env") + +# Workspace root — read from config.yaml (dataset.workspace_root) +_WORKSPACE_ROOT_DEFAULT = "benchmark/memory_workspaces/beam" + +# --------------------------------------------------------------------------- +# Logging +# --------------------------------------------------------------------------- +_DEFAULT_LOG_FORMAT = "%(asctime)s | %(levelname)s | %(message)s" + +logging.basicConfig(level=logging.INFO, format=_DEFAULT_LOG_FORMAT) +logger = logging.getLogger("beam") + +# Noisy library loggers silenced by default +_NOISY_LOGGERS = [ + "httpx", + "httpcore", + "openai", + "uvicorn", + "multipart", + "asyncio", + "watchfiles", + "filelock", +] + + +def setup_logging( + log_level: str, + reme_log_level: str, + log_dir: str | None = None, +): + """Configure logging for the eval runner and reme internals. + + Args: + log_level: Level for the eval runner logger (DEBUG/INFO/WARNING/ERROR). + reme_log_level: Level for reme's internal loguru logger. + log_dir: Per-run log directory (absolute path). None = no file logging. + """ + numeric = getattr(logging, log_level.upper(), logging.INFO) + # Eval runner logger + logging.getLogger().setLevel(numeric) + logger.setLevel(numeric) + + # Suppress noisy library loggers when above DEBUG + if numeric > logging.DEBUG: + for name in _NOISY_LOGGERS: + lib_logger = logging.getLogger(name) + lib_logger.setLevel(max(numeric, logging.WARNING)) + + # Add file handler for eval runner if log_dir is specified + if log_dir: + os.makedirs(log_dir, exist_ok=True) + log_filepath = os.path.join(log_dir, "runner.log") + file_handler = logging.FileHandler(log_filepath, encoding="utf-8") + file_handler.setLevel(numeric) + file_handler.setFormatter(logging.Formatter(_DEFAULT_LOG_FORMAT)) + logging.getLogger().addHandler(file_handler) + logger.info(f"Eval runner log file: {log_filepath}") + + # Reme internal logger (loguru) — will be applied per-worker via _configure_worker + os.environ["REME_LOG_LEVEL"] = reme_log_level.upper() + if log_dir: + os.environ["REME_LOG_DIR"] = log_dir + + +def _configure_worker( + log_level: str, + reme_log_level: str, + log_dir: str | None = None, +): + """Set up logging inside a multiprocessing worker process. + + Must be called at the top of each worker because child processes inherit + parent state but loguru sinks are NOT shared across fork/spawn. + """ + numeric = getattr(logging, log_level.upper(), logging.INFO) + logging.basicConfig(level=numeric, format=_DEFAULT_LOG_FORMAT, force=True) + logging.getLogger("beam").setLevel(numeric) + if numeric > logging.DEBUG: + for name in _NOISY_LOGGERS: + logging.getLogger(name).setLevel(max(numeric, logging.WARNING)) + + # Add file handler for eval runner in worker process + if log_dir: + os.makedirs(log_dir, exist_ok=True) + pid = os.getpid() + log_filepath = os.path.join(log_dir, f"worker-{pid}.log") + file_handler = logging.FileHandler(log_filepath, encoding="utf-8") + file_handler.setLevel(numeric) + file_handler.setFormatter(logging.Formatter(_DEFAULT_LOG_FORMAT)) + logging.getLogger().addHandler(file_handler) + + # Re-initialize loguru for reme internals at the desired level + from reme.utils import get_logger + + reme_log_dir = log_dir or "logs" + get_logger(log_dir=reme_log_dir, level=reme_log_level.upper(), force_init=True) + + +# --------------------------------------------------------------------------- +# Config loading +# --------------------------------------------------------------------------- +def load_eval_config(config_path: str | None = None) -> dict: + """Load evaluation config yaml with env-var expansion.""" + if config_path is None: + config_path = str(Path(__file__).parent / "config.yaml") + with open(config_path, encoding="utf-8") as f: + raw = f.read() + + # Expand ${VAR} and ${VAR:-default} + def _expand(m): + expr = m.group(1) + if ":-" in expr: + key, default = expr.split(":-", 1) + return os.environ.get(key, default) + return os.environ.get(expr, "") + + raw = re.sub(r"\$\{([^}]+)\}", _expand, raw) + return yaml.safe_load(raw) + + +# --------------------------------------------------------------------------- +# BEAM data loading +# --------------------------------------------------------------------------- +def parse_beam_time_anchor(time_str: str) -> datetime: + """Parse BEAM time_anchor format: 'March-15-2024' -> datetime.""" + for fmt in ("%B-%d-%Y", "%b-%d-%Y"): + try: + return datetime.strptime(time_str, fmt) + except ValueError: + continue + raise ValueError(f"Cannot parse time_anchor: {time_str!r}") + + +def load_beam_chat(chat_path: Path, chat_size: str, case_id: str) -> list[dict]: + """Load BEAM chat.json and convert to ReMe session format. + + Each batch becomes one session with all its turns flattened. + Each turn resolves its own time_anchor independently; turns without + an explicit time_anchor inherit from the most recent preceding turn. + Returns list of sessions, each with: + - session_id: str + - date: str (YYYY-MM-DD) — derived from the *first* turn's time + - messages: list[dict] with name, role, content, created_at + """ + with open(chat_path, encoding="utf-8") as f: + batches = json.load(f) + + sessions = [] + for batch in batches: + batch_num = batch["batch_number"] + + # Resolve batch-level fallback (used when no turn has a time_anchor) + batch_anchor = batch.get("time_anchor") + if not batch_anchor: + batch_anchor = "January-1-2024" + + # Flatten all turns, resolving time_anchor per turn + messages = [] + prev_dt = None # carries forward from previous turn + first_dt = None # for session-level date + + for turn in batch["turns"]: + # Find this turn's own time_anchor from its messages + turn_anchor = None + for msg in turn: + if msg.get("time_anchor"): + turn_anchor = msg["time_anchor"] + break + + if turn_anchor: + dt = parse_beam_time_anchor(turn_anchor) + elif prev_dt is not None: + dt = prev_dt # inherit from previous turn + else: + dt = parse_beam_time_anchor(batch_anchor) + + if first_dt is None: + first_dt = dt + prev_dt = dt + + for msg in turn: + role = msg["role"] + messages.append( + { + "name": role, + "role": role, + "content": msg["content"], + "created_at": dt.strftime("%Y-%m-%dT%H:%M:%S"), + }, + ) + + sessions.append( + { + "session_id": f"beam_{chat_size}_{case_id}_batch{batch_num}", + "date": first_dt.strftime("%Y-%m-%d"), + "messages": messages, + }, + ) + + return sessions + + +def get_available_cases(beam_root: Path, chat_size: str) -> list[str]: + """Return sorted list of case IDs for a given chat size.""" + chats_dir = beam_root / "chats" / chat_size + if not chats_dir.exists(): + return [] + return sorted( + [d.name for d in chats_dir.iterdir() if d.is_dir()], + key=int, + ) + + +# --------------------------------------------------------------------------- +# Answer generation +# --------------------------------------------------------------------------- +async def answer_question_agentic(app, question: str) -> tuple[str, dict]: + """Answer a probing question using ReMe's agentic_answer job. + + Returns (answer, metadata) + """ + query_resp = await app.run_job( + "agentic_answer", + query=question, + ) + answer = (query_resp.answer or "").strip() + + return answer, {"mode": "agentic"} + + +# --------------------------------------------------------------------------- +# BEAM rubric-based LLM-as-Judge +# --------------------------------------------------------------------------- +async def judge_answer( + app, + question: str, + llm_response: str, + rubric: list[str], + question_type: str = "", +) -> dict: + """Judge an answer via the answer_judge job (beam_rubric_judge_step).""" + judge_resp = await app.run_job( + "answer_judge", + llm_response=llm_response, + rubric=rubric, + probing_question=question, + question_type=question_type, + ) + result = { + "llm_judge_score": (judge_resp.metadata or {}).get("llm_judge_score", 0.0), + "llm_judge_responses": (judge_resp.metadata or {}).get("llm_judge_responses", []), + } + # Include event_ordering extra metrics if present + eo = (judge_resp.metadata or {}).get("event_ordering") + if eo: + result["event_ordering"] = eo + return result + + +# --------------------------------------------------------------------------- +# Main evaluation pipeline +# --------------------------------------------------------------------------- +async def evaluate_case(eval_config: dict, case_id: str, eval_only: bool = False) -> dict: + """Evaluate a single BEAM case end-to-end. + + Args: + eval_config: The evaluation configuration dict. + case_id: The case directory name (e.g. "1"). + eval_only: If True, skip ingestion and only run query+judge + using the existing workspace. + + Returns: + A results dict with all questions, answers, and judgments. + """ + from reme import Application + from reme.config import resolve_app_config + + dataset_cfg = eval_config["dataset"] + chat_size = dataset_cfg["chat_size"] + beam_root = _PROJECT_ROOT / dataset_cfg.get("beam_root", "benchmark/datasets/BEAM") + chat_path = beam_root / "chats" / chat_size / case_id / "chat.json" + probing_questions_path = beam_root / "chats" / chat_size / case_id / "probing_questions" / "probing_questions.json" + + if not chat_path.exists(): + raise FileNotFoundError(f"Chat file not found: {chat_path}") + if not probing_questions_path.exists(): + raise FileNotFoundError(f"Probing questions not found: {probing_questions_path}") + + logger.info( + "[Case %s] size=%s%s", + case_id, + chat_size, + " [eval_only]" if eval_only else "", + ) + + # Workspace setup + workspace_root = _PROJECT_ROOT / dataset_cfg.get("workspace_root", _WORKSPACE_ROOT_DEFAULT) + case_dir = workspace_root / f"{chat_size}_{case_id}" + workspace_dir = str(case_dir / ".reme") + + if eval_only: + if not case_dir.exists() or not Path(workspace_dir).exists(): + raise FileNotFoundError( + f"[Case {case_id}] eval_only: workspace not found at {case_dir}. " + f"Run without --eval_only first to build the workspace.", + ) + else: + if case_dir.exists(): + shutil.rmtree(case_dir) + logger.info(f"[Case {case_id}] Cleaned existing workspace: {case_dir}") + else: + logger.info(f"[Case {case_id}] Workspace not found, creating: {case_dir}") + case_dir.mkdir(parents=True, exist_ok=True) + + # Pre-initialize ReMe's loguru logger with the correct log_dir + output_cfg = eval_config.get("output", {}) + if output_cfg.get("log_to_file", False): + reme_log_dir = os.environ.get("REME_LOG_DIR") + if reme_log_dir: + from reme.utils import get_logger + + get_logger( + log_dir=reme_log_dir, + level=os.environ.get("REME_LOG_LEVEL", "INFO"), + log_to_console=output_cfg.get("log_to_console", True), + log_to_file=True, + force_init=True, + ) + + cfg = resolve_app_config( + config=eval_config["reme"]["config"], + workspace_dir=workspace_dir, + log_to_console=output_cfg.get("log_to_console", True), + log_to_file=output_cfg.get("log_to_file", False), + enable_logo=False, + ) + + app = Application(**cfg) + await app.start() + + sessions_ingested = 0 + try: + if not eval_only: + # ── Phase 1: Ingest sessions ────────────────────────────── + sessions = load_beam_chat(chat_path, chat_size, case_id) + logger.info(f"[Case {case_id}] Loaded {len(sessions)} sessions from chat.json") + + for i, session in enumerate(sessions): + logger.info( + f"[Case {case_id}] Ingesting session {i+1}/{len(sessions)}: " + f"id={session['session_id']} date={session['date']} " + f"msgs={len(session['messages'])}", + ) + resp = await app.run_job( + "auto_memory", + messages=session["messages"], + session_id=session["session_id"], + date=session["date"], + ) + if not resp.success: + logger.warning(f"[Case {case_id}] auto_memory failed: {resp.answer}") + else: + logger.info( + f"[Case {case_id}] auto_memory success: " f"{resp.answer[:100] if resp.answer else ''}", + ) + await app.run_job("index_update") + sessions_ingested += 1 + + # Final digest update + logger.info(f"[Case {case_id}] Running digest_update...") + await app.run_job("digest_update") + logger.info(f"[Case {case_id}] Ingestion complete.") + + # ── Phase 2: Answer + Judge probing questions ─────────────── + with open(probing_questions_path, encoding="utf-8") as f: + probing_questions = json.load(f) + + total_questions = sum(len(v) for v in probing_questions.values()) + logger.info(f"[Case {case_id}] Total probing questions: {total_questions}") + + all_question_results = [] + q_idx = 0 + + for q_type in probing_questions: + logger.info( + f"[Case {case_id}] Question type: {q_type} " f"({len(probing_questions[q_type])} questions)", + ) + + for i, q in enumerate(probing_questions[q_type]): + q_idx += 1 + question = q["question"] + rubric = q.get("rubric", []) + logger.info( + f"[Case {case_id}] [{q_idx}/{total_questions}] " f"{q_type} Q{i+1}: {question[:100]}...", + ) + + q_result = { + "question_type": q_type, + "question_index": i, + "question": question, + "rubric": rubric, + } + + # Agentic answer + try: + agentic_answer, agentic_meta = await answer_question_agentic( + app, + question, + ) + except Exception as e: + logger.error(f"[Case {case_id}] Agentic answer failed: {e}") + agentic_answer = f"(error: {e})" + agentic_meta = {"error": str(e)} + + if not agentic_answer: + agentic_answer = "(no answer generated)" + logger.info(f"[Case {case_id}] Agentic answer: {agentic_answer[:200]}...") + + # Judge agentic answer + logger.info(f"[Case {case_id}] Judging agentic ({q_type})...") + agentic_judgment = await judge_answer( + app, + question, + agentic_answer, + rubric, + question_type=q_type, + ) + logger.info( + f"[Case {case_id}] Agentic score: " f"{agentic_judgment['llm_judge_score']:.3f}", + ) + + q_result["agentic_response"] = agentic_answer + q_result["agentic_judgment"] = agentic_judgment + q_result["agentic_metadata"] = agentic_meta + + all_question_results.append(q_result) + + finally: + await app.close() + + return { + "case_id": case_id, + "chat_size": chat_size, + "sessions_ingested": sessions_ingested, + "total_questions": len(all_question_results), + "questions": all_question_results, + } + + +# --------------------------------------------------------------------------- +# Worker: runs a single case in its own process with its own event loop +# --------------------------------------------------------------------------- +def _evaluate_case_worker(task_input: tuple) -> dict: + """Worker function for multiprocessing. Each process gets its own event loop.""" + eval_config, case_id, log_level, reme_log_level, eval_only, log_dir = task_input + import asyncio # pylint: disable=import-outside-toplevel + + _configure_worker(log_level, reme_log_level, log_dir=log_dir) + + # Suppress httpx GC noise + logging.getLogger("asyncio").setLevel(logging.CRITICAL) + + return asyncio.run(evaluate_case(eval_config, case_id, eval_only=eval_only)) + + +def _indexed_worker(indexed_input: tuple) -> tuple: + """Module-level wrapper for imap_unordered with index tracking.""" + idx, task_input = indexed_input + return idx, _evaluate_case_worker(task_input) + + +def _resolve_num_workers(configured: int) -> int: + """Resolve num_workers: 0=auto (cpu_count-2, min 1), 1=sequential, >1=parallel.""" + if configured == 0: + return max(1, (os.cpu_count() or 4) - 2) + return max(1, configured) + + +# --------------------------------------------------------------------------- +# Entry point +# --------------------------------------------------------------------------- +def main( # pylint: disable=too-many-statements + config_path: str | None = None, + log_level: str = "INFO", + reme_log_level: str = "INFO", + eval_only: bool = False, +): + """Run the BEAM evaluation pipeline. + + Args: + config_path: Path to the YAML config file. + log_level: Log level for the eval runner. + reme_log_level: Log level for reme internal logs. + eval_only: If True, skip ingestion and only run query+judge using + existing workspaces. + """ + from multiprocessing import Pool # pylint: disable=import-outside-toplevel + + # Load config BEFORE logging setup so log_dir is available + eval_config = load_eval_config(config_path) + + # Resolve per-run log directory from config + output_cfg = eval_config.get("output", {}) + log_dir_abs = None + if output_cfg.get("log_to_file", False): + log_dir_raw = output_cfg.get("log_dir", "logs") + log_prefix = output_cfg.get("log_prefix", "beam") + run_ts = datetime.now().strftime("%Y-%m-%d_%H-%M-%S") + log_dir_abs = str(_PROJECT_ROOT / log_dir_raw / f"{log_prefix}_{run_ts}") + + setup_logging(log_level, reme_log_level, log_dir=log_dir_abs) + dataset_cfg = eval_config["dataset"] + chat_size = dataset_cfg["chat_size"] + beam_root = _PROJECT_ROOT / dataset_cfg.get("beam_root", "benchmark/datasets/BEAM") + + # Determine which cases to run + case_ids = dataset_cfg.get("case_ids") or [] + if not case_ids: + case_ids = get_available_cases(beam_root, chat_size) + + # Pagination + start = dataset_cfg.get("start_index", 0) + num_items = dataset_cfg.get("num_items", 0) + if num_items > 0: + case_ids = case_ids[start : start + num_items] + elif start > 0: + case_ids = case_ids[start:] + + if not case_ids: + logger.error(f"No cases found for chat_size={chat_size}") + return + + logger.info( + "Evaluating %d case(s) for chat_size=%s: %s%s", + len(case_ids), + chat_size, + case_ids, + " [eval_only: query+judge only]" if eval_only else "", + ) + + # Resolve parallelism + num_workers = _resolve_num_workers(eval_config["evaluation"].get("num_workers", 1)) + logger.info(f"Using {num_workers} worker(s)") + + # Create output directory + output_dir = _PROJECT_ROOT / output_cfg.get("dir", "benchmark/results/beam") + output_dir.mkdir(parents=True, exist_ok=True) + + # Create workspace root directory + workspace_root = _PROJECT_ROOT / dataset_cfg.get("workspace_root", _WORKSPACE_ROOT_DEFAULT) + workspace_root.mkdir(parents=True, exist_ok=True) + + # Pre-check: verify all workspaces exist in eval_only mode + if eval_only: + missing_cases = [] + for case_id in case_ids: + case_dir = workspace_root / f"{chat_size}_{case_id}" + if not case_dir.exists() or not (case_dir / ".reme").exists(): + missing_cases.append(case_id) + if missing_cases: + preview = missing_cases[:10] + suffix = "..." if len(missing_cases) > 10 else "" + raise FileNotFoundError( + f"eval_only: {len(missing_cases)} workspace(s) not found under {workspace_root}. " + f"Missing cases: {preview}{suffix}. " + f"Run without --eval_only first to build the workspaces.", + ) + + # Build task args + task_args = [(eval_config, case_id, log_level, reme_log_level, eval_only, log_dir_abs) for case_id in case_ids] + + # Progress tracking + total_items = len(task_args) + completed_count = [0] + start_time = time.time() + progress_lock = threading.Lock() + + def _print_progress(prefix: str = "PROGRESS"): + elapsed = time.time() - start_time + elapsed_min = elapsed / 60 + done = completed_count[0] + pct = 100.0 * done / total_items if total_items else 0 + eta_str = "N/A" + if done > 0: + eta_sec = elapsed / done * (total_items - done) + eta_str = f"{eta_sec/60:.1f}min" + print( + f"[{prefix}] {datetime.now().strftime('%Y-%m-%d %H:%M:%S')} | " + f"{done}/{total_items} ({pct:.1f}%) completed | " + f"elapsed={elapsed_min:.1f}min | ETA={eta_str}", + flush=True, + ) + + def _progress_timer(): + """Background thread: print progress every 10 minutes.""" + while not _timer_stop.is_set(): + _timer_stop.wait(600) + if not _timer_stop.is_set(): + with progress_lock: + _print_progress() + + _timer_stop = threading.Event() + timer_thread = threading.Thread(target=_progress_timer, daemon=True) + timer_thread.start() + + # Run evaluation + if num_workers == 1: + results = [] + for task_input in task_args: + result = _evaluate_case_worker(task_input) + results.append(result) + with progress_lock: + completed_count[0] += 1 + else: + results = [None] * total_items + indexed_args = list(enumerate(task_args)) + + with Pool(processes=num_workers) as pool: + for idx, result in pool.imap_unordered(_indexed_worker, indexed_args): + results[idx] = result + with progress_lock: + completed_count[0] += 1 + + # Stop progress timer + _timer_stop.set() + timer_thread.join(timeout=2) + + # Save results + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + output_file = output_dir / f"results_{chat_size}_{timestamp}.json" + with open(output_file, "w", encoding="utf-8") as f: + json.dump(results, f, ensure_ascii=False, indent=2) + logger.info(f"Results saved to {output_file}") + + # Final progress + _print_progress("FINAL") + + # Print concise summary + print("\n" + "=" * 70) + print(f" BEAM EVALUATION RESULTS | size={chat_size} cases={len(results)}") + print("=" * 70) + + # Per-type stats (agentic only) + type_scores: dict[str, list[float]] = {} + type_binary_scores: dict[str, list[float]] = {} + all_scores: list[float] = [] + all_binary_scores: list[float] = [] + + for case_result in results: + if "error" in case_result: + continue + for q in case_result.get("questions", []): + judgment = q.get("agentic_judgment", {}) + score = judgment.get("llm_judge_score", 0.0) + # Binary: convert each rubric item score to 0/1, then average + judge_responses = judgment.get("llm_judge_responses", []) + if judge_responses: + binary_scores_per_item = [1.0 if r.get("score", 0) >= 1.0 else 0.0 for r in judge_responses] + binary_score = sum(binary_scores_per_item) / len(binary_scores_per_item) + else: + binary_score = 1.0 if score > 0.99 else 0.0 + qtype = q["question_type"] + if qtype not in type_scores: + type_scores[qtype] = [] + type_binary_scores[qtype] = [] + type_scores[qtype].append(score) + type_binary_scores[qtype].append(binary_score) + all_scores.append(score) + all_binary_scores.append(binary_score) + + print("\n ── AGENTIC ──") + if all_scores: + for qtype in sorted(type_scores.keys()): + scores = type_scores[qtype] + avg = sum(scores) / len(scores) if scores else 0 + bin_scores = type_binary_scores[qtype] + bin_avg = sum(bin_scores) / len(bin_scores) if bin_scores else 0 + print(f" {qtype:<40s}: {avg:.3f} binary={bin_avg:.3f} ({len(scores)} Qs)") + overall = sum(all_scores) / len(all_scores) if all_scores else 0 + binary_overall = sum(all_binary_scores) / len(all_binary_scores) if all_binary_scores else 0 + print(f" {'-'*38}") + print(f" {'OVERALL':<40s}: {overall:.3f} binary={binary_overall:.3f} ({len(all_scores)} Qs)") + else: + print(" (no results)") + + # Per-case summary + print("\n ── Per-Case Summary ──") + for case_result in results: + case_id = case_result["case_id"] + if "error" in case_result: + print(f" Case {case_id}: ERROR — {case_result['error']}") + continue + n_qs = case_result.get("total_questions", 0) + n_sessions = case_result.get("sessions_ingested", 0) + parts = [f"Case {case_id}: {n_sessions} sessions, {n_qs} questions"] + questions = case_result.get("questions", []) + scores = [q.get("agentic_judgment", {}).get("llm_judge_score", 0.0) for q in questions] + if scores: + avg = sum(scores) / len(scores) + # Binary: 0/1 per rubric item, average per question, then across questions + bin_scores = [] + for q in questions: + judge_responses = q.get("agentic_judgment", {}).get("llm_judge_responses", []) + if judge_responses: + item_bins = [1.0 if r.get("score", 0) >= 1.0 else 0.0 for r in judge_responses] + bin_scores.append(sum(item_bins) / len(item_bins)) + else: + s = q.get("agentic_judgment", {}).get("llm_judge_score", 0.0) + bin_scores.append(1.0 if s > 0.99 else 0.0) + bin_avg = sum(bin_scores) / len(bin_scores) + parts.append(f"agentic={avg:.3f} binary={bin_avg:.3f}") + print(f" {' | '.join(parts)}") + + print("=" * 70) + total_elapsed = time.time() - start_time + print(f"\n Total time: {total_elapsed/60:.1f} min") + print("\n" + "=" * 70) + print(" [DONE] BEAM EVALUATION COMPLETED SUCCESSFULLY") + print("=" * 70 + "\n") + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser(description="BEAM evaluation runner") + parser.add_argument("--config", type=str, default=None, help="Path to config.yaml") + parser.add_argument( + "--log-level", + type=str, + default="INFO", + choices=["DEBUG", "INFO", "WARNING", "ERROR"], + help="Log level for the eval runner (default: INFO)", + ) + parser.add_argument( + "--reme-log-level", + type=str, + default="INFO", + choices=["DEBUG", "INFO", "WARNING", "ERROR"], + help="Log level for reme internal logs — loguru (default: INFO)", + ) + parser.add_argument( + "-q", + "--quiet", + action="store_true", + help="Shortcut for --log-level WARNING --reme-log-level WARNING", + ) + parser.add_argument( + "--eval_only", + action="store_true", + help="Skip ingestion. Reuse existing workspaces and only run query+judge.", + ) + args = parser.parse_args() + + if args.quiet: + args.log_level = "WARNING" + args.reme_log_level = "WARNING" + + main(args.config, args.log_level, args.reme_log_level, eval_only=args.eval_only) diff --git a/benchmark/datasets/README.md b/benchmark/datasets/README.md new file mode 100644 index 00000000..3a772a9d --- /dev/null +++ b/benchmark/datasets/README.md @@ -0,0 +1,40 @@ +# 数据集下载说明 + +本目录包含 ReMe 评测所需的数据集。部分数据集体积较大，不纳入 Git 版本管理，需要手动下载。 + +## LongMemEval（cleaned-S） + +ReMe 仅使用 LongMemEval 的 **cleaned-S** 版本，数据托管在 HuggingFace： +[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2) +（下载脚本经 hf-mirror.com 镜像源获取）。 + +按以下步骤下载： + +```bash +cd benchmark/datasets/longmemeval + +# 下载 cleaned-S 数据文件（已存在则自动跳过） +python download.py +``` + +下载完成后，目录下应包含以下文件： + +| 文件名 | 说明 | +| --- | --- | +| `longmemeval_s_reme_cleaned.json` | cleaned-S 数据集，已包含 ground truth 字段 | +| `download.py` | 下载脚本（已随仓库提供） | + +> **注意**：下载脚本使用 hf-mirror.com 镜像源，如需更换源请修改 `download.py` 中的 `BASE_URL`。 + +下载完成后即可参照 [`benchmark/README_ZH.md`](../README_ZH.md) 运行 LongMemEval 评测。 + +## BEAM + +BEAM 数据集为公开仓库，直接 clone 到 `benchmark/datasets/` 目录下即可： + +```bash +cd benchmark/datasets +git clone https://github.com/mohammadtavakoli78/BEAM.git +``` + +clone 完成后，`benchmark/datasets/BEAM/` 目录下应包含 `chats/`、`src/`、`topics/` 等子目录。 diff --git a/benchmark/datasets/README_EN.md b/benchmark/datasets/README_EN.md new file mode 100644 index 00000000..24a7bab0 --- /dev/null +++ b/benchmark/datasets/README_EN.md @@ -0,0 +1,40 @@ +# Dataset Download Guide + +This directory contains datasets required for ReMe evaluation. Some datasets are large and excluded from Git version control — they must be downloaded manually. + +## LongMemEval (cleaned-S) + +ReMe uses only the **cleaned-S** split of LongMemEval, hosted on HuggingFace: +[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2) +(the script downloads via the hf-mirror.com mirror). + +Download it with: + +```bash +cd benchmark/datasets/longmemeval + +# Download the cleaned-S data file (skipped automatically if it already exists) +python download.py +``` + +After downloading, the directory should contain: + +| File | Description | +| --- | --- | +| `longmemeval_s_reme_cleaned.json` | cleaned-S dataset with ground truth fields included | +| `download.py` | Download script (included in repo) | + +> **Note**: The download script uses hf-mirror.com by default. To use a different mirror, modify `BASE_URL` in `download.py`. + +Once the download completes, follow [`benchmark/README.md`](../README.md) to run the LongMemEval evaluation. + +## BEAM + +BEAM is a public repository. Clone it directly into the `benchmark/datasets/` directory: + +```bash +cd benchmark/datasets +git clone https://github.com/mohammadtavakoli78/BEAM.git +``` + +After cloning, `benchmark/datasets/BEAM/` should contain `chats/`, `src/`, `topics/` and other subdirectories. diff --git a/benchmark/datasets/longmemeval/download.py b/benchmark/datasets/longmemeval/download.py new file mode 100644 index 00000000..d6754640 --- /dev/null +++ b/benchmark/datasets/longmemeval/download.py @@ -0,0 +1,65 @@ +"""Download the LongMemEval cleaned-S dataset used by ReMe. + +Source: https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2 +(downloaded via the hf-mirror.com mirror for reliability). + +The file ``longmemeval_s_reme_cleaned.json`` is saved under this directory using the same +name as on the remote (``benchmark/longmemeval/config.yaml`` points to it). + +Usage: + python download.py # download cleaned-S (skip if it already exists) +""" + +import os +import sys +import urllib.request + +BASE_URL = "https://hf-mirror.com/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2/resolve/main" +TARGET_DIR = os.path.dirname(os.path.abspath(__file__)) + +# Files to download (saved with the same name as on the remote). +FILES = [ + "longmemeval_s_reme_cleaned.json", +] + + +def download_file(filename: str): + """Download a single file from the mirror to the target directory.""" + url = f"{BASE_URL}/{filename}" + dest = os.path.join(TARGET_DIR, filename) + + if os.path.exists(dest): + size = os.path.getsize(dest) + print(f" [skip] {filename} already exists ({size / 1024 / 1024:.1f} MB)") + return + + print(f" [downloading] {filename} ...") + try: + urllib.request.urlretrieve(url, dest, reporthook=_progress) + size = os.path.getsize(dest) + print(f"\n [done] {filename} ({size / 1024 / 1024:.1f} MB)") + except Exception as e: + print(f"\n [error] {filename}: {e}") + if os.path.exists(dest): + os.remove(dest) + sys.exit(1) + + +def _progress(block_num, block_size, total_size): + downloaded = block_num * block_size + if total_size > 0: + pct = min(100, downloaded * 100 / total_size) + mb = downloaded / 1024 / 1024 + total_mb = total_size / 1024 / 1024 + sys.stdout.write(f"\r {mb:.1f}/{total_mb:.1f} MB ({pct:.1f}%)") + else: + mb = downloaded / 1024 / 1024 + sys.stdout.write(f"\r {mb:.1f} MB downloaded") + sys.stdout.flush() + + +if __name__ == "__main__": + print(f"Downloading LongMemEval cleaned-S dataset to: {TARGET_DIR}\n") + for fname in FILES: + download_file(fname) + print("\nAll files downloaded successfully!") diff --git a/benchmark/kill.sh b/benchmark/kill.sh new file mode 100644 index 00000000..9f4a5237 --- /dev/null +++ b/benchmark/kill.sh @@ -0,0 +1,76 @@ +#!/bin/bash +# 杀死指定进程及其所有子进程 +# Usage: bash kill.sh + +if [ -z "$1" ]; then + echo "Usage: bash kill.sh " + echo " 杀死指定进程及其所有子进程" + exit 1 +fi + +PID=$1 + +# 检查进程是否存在 +if ! kill -0 "$PID" 2>/dev/null; then + echo "进程 $PID 不存在" + exit 1 +fi + +# 递归收集所有子进程（包括子进程的子进程） +collect_children() { + local parent=$1 + local children + children=$(ps -o pid= --ppid "$parent" 2>/dev/null | tr -d ' ') + for child in $children; do + collect_children "$child" + done + echo "$parent" +} + +# 收集进程树（子进程在前，父进程在后，保证先杀子再杀父） +PROCESS_TREE=$(collect_children "$PID") +TOTAL=$(echo "$PROCESS_TREE" | wc -l | tr -d ' ') + +echo "进程树（共 $TOTAL 个进程）:" +while read -r p; do + cmd=$(ps -o args= -p "$p" 2>/dev/null | head -c 80) + printf " PID=%-8s %s\n" "$p" "$cmd" +done <<< "$PROCESS_TREE" + +# 先 SIGTERM 优雅终止 +echo "" +echo "发送 SIGTERM..." +while read -r p; do + kill "$p" 2>/dev/null +done <<< "$PROCESS_TREE" + +# 等待最多 5 秒 +for i in $(seq 1 5); do + alive=false + while read -r p; do + if kill -0 "$p" 2>/dev/null; then + alive=true + fi + done <<< "$PROCESS_TREE" + if [ "$alive" = false ]; then + break + fi + sleep 1 +done + +# 检查是否还有残留，强制 SIGKILL +remaining=false +while read -r p; do + if kill -0 "$p" 2>/dev/null; then + remaining=true + fi +done <<< "$PROCESS_TREE" + +if [ "$remaining" = true ]; then + echo "部分进程未响应，发送 SIGKILL..." + while read -r p; do + kill -9 "$p" 2>/dev/null + done <<< "$PROCESS_TREE" +fi + +echo "已终止进程树（根 PID=$PID，共 $TOTAL 个进程）" diff --git a/benchmark/longmemeval/clean_sample_outputs.py b/benchmark/longmemeval/clean_sample_outputs.py deleted file mode 100644 index 4fcd50ae..00000000 --- a/benchmark/longmemeval/clean_sample_outputs.py +++ /dev/null @@ -1,141 +0,0 @@ -#!/usr/bin/env python3 -"""Remove generated LongMemEval files while keeping source inputs. - -For each ``datasets/longmemeval/`` workspace, this keeps only: - - query.json - - answer.json - - session/ - -All other files or directories in the sample root are considered generated -artifacts and can be removed. AppleDouble files whose names start with ``._`` -are also removed recursively, including under ``session/``. The script is -dry-run by default; pass ``--apply`` to actually delete. To delete only specific -root-level generated files, pass one or more ``--filename`` values. - -Examples: - python benchmark/longmemeval/clean_sample_outputs.py - python benchmark/longmemeval/clean_sample_outputs.py --apply - python benchmark/longmemeval/clean_sample_outputs.py --start 36 --end 79 --apply - python benchmark/longmemeval/clean_sample_outputs.py --filename check_golden.json --apply - python benchmark/longmemeval/clean_sample_outputs.py --filename session_review.json --apply -""" - -import argparse -import shutil -import time -from collections.abc import Iterator -from pathlib import Path - -REPO = Path(__file__).resolve().parents[2] -DATA = REPO / "datasets" / "longmemeval" -KEEP = {"query.json", "answer.json", "session"} - - -def parse_args() -> argparse.Namespace: - """Parse command-line arguments.""" - p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - p.add_argument("--start", type=int, default=0, help="first numeric sample id to clean, inclusive (default 0)") - p.add_argument("--end", type=int, default=499, help="last numeric sample id to clean, inclusive (default 499)") - p.add_argument("--limit", type=int, default=0, help="only clean the first N selected samples (0 = all)") - p.add_argument("--progress-every", type=int, default=25, help="print progress every N samples when applying") - p.add_argument( - "--filename", - action="append", - default=[], - help="delete only this root-level file or directory name; can be passed multiple times", - ) - p.add_argument("--apply", action="store_true", help="actually delete files; default is dry-run") - return p.parse_args() - - -def sample_ids() -> list[str]: - """List all numeric sample IDs.""" - ids = [p.name for p in DATA.iterdir() if p.is_dir() and p.name.isdigit()] - return sorted(ids, key=int) - - -def delete_path(path: Path) -> None: - """Delete a file, symlink, or directory.""" - if path.is_dir() and not path.is_symlink(): - shutil.rmtree(path) - else: - path.unlink() - - -def iter_sample_targets(sample_dir: Path, filenames: set[str] | None = None) -> Iterator[Path]: - """Yield generated artifacts for one sample. - - Root-level generated directories are yielded as a whole, so there is no - need to recurse into them. AppleDouble files are only searched inside the - kept ``session/`` directory. - """ - if filenames: - for name in sorted(filenames): - path = sample_dir / name - if path.exists(): - yield path - return - - for path in sorted(sample_dir.iterdir(), key=lambda p: p.name): - if path.name not in KEEP: - yield path - - session_dir = sample_dir / "session" - if session_dir.is_dir(): - yield from session_dir.rglob("._*") - - -def main() -> int: - """Main entry point.""" - args = parse_args() - if args.end < args.start: - raise ValueError(f"--end ({args.end}) must be >= --start ({args.start})") - filenames = {name.strip() for name in args.filename if name.strip()} - invalid_filenames = [name for name in filenames if Path(name).name != name] - if invalid_filenames: - raise ValueError(f"--filename only accepts root-level names, got: {invalid_filenames}") - - ids = [idx for idx in sample_ids() if args.start <= int(idx) <= args.end] - if args.limit: - ids = ids[: args.limit] - - total_targets = 0 - deleted = 0 - started_at = time.time() - for ordinal, idx in enumerate(ids, start=1): - sample_dir = DATA / idx - sample_started_at = time.time() - targets = list(iter_sample_targets(sample_dir, filenames=filenames)) - total_targets += len(targets) - print(f"[sample {ordinal}/{len(ids)}] {idx} targets={len(targets)}", flush=True) - for path in targets: - if args.apply: - target_started_at = time.time() - print(f"[delete] {path}", flush=True) - delete_path(path) - deleted += 1 - print(f"[deleted] {path} elapsed={time.time() - target_started_at:.1f}s", flush=True) - else: - print(f"[would-delete] {path}") - if args.apply and args.progress_every > 0 and (int(idx) + 1) % args.progress_every == 0: - elapsed = time.time() - started_at - print( - f"[progress] processed={ordinal}/{len(ids)} through={idx} " f"deleted={deleted} elapsed={elapsed:.1f}s", - flush=True, - ) - print(f"[sample-done] {idx} elapsed={time.time() - sample_started_at:.1f}s", flush=True) - - mode = "DELETE" if args.apply else "DRY-RUN" - print( - f"{mode} LongMemEval generated artifacts: samples={len(ids)} " - f"targets={total_targets} deleted={deleted if args.apply else 0} range={args.start}..{args.end}", - flush=True, - ) - - if not args.apply: - print("No files deleted. Re-run with --apply to delete these paths.", flush=True) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/benchmark/longmemeval/config.yaml b/benchmark/longmemeval/config.yaml new file mode 100644 index 00000000..bc62d7a7 --- /dev/null +++ b/benchmark/longmemeval/config.yaml @@ -0,0 +1,32 @@ +# LongMemEval evaluation configuration +# This file controls what/how to evaluate. + +dataset: + path: "benchmark/datasets/longmemeval/longmemeval_s_reme_cleaned.json" + start_index: 0 # first item index + num_items: 500 # how many items to evaluate (starting from start_index) + max_sessions: 0 # 0 = all sessions; >0 = limit sessions per item for testing + question_types: [] # filter by question_type; empty list = no filtering (all types) + workspace_root: "benchmark/memory_workspaces/longmemeval-s" # workspace root for item workspaces + +evaluation: + # LLM-as-judge uses the 'judge' as_llm component defined in lme.yaml + # Model and credentials are configured there (reading from .env) + # Judgment is always binary (yes/no) — defined in lme/llm_judge.yaml + num_workers: 32 # 0 = auto (cpu_count - 2, min 1); 1 = sequential; >1 = parallel + filter_future_sessions: true # true = only ingest sessions with timestamp <= question_date + +reme: + config: "lme.yaml" # reme config to use (in reme/config/) + # Dream trigger: when gap between consecutive sessions crosses this hour (23:00) + dream_trigger_hour: 23 + # Dream scan_days for each trigger + dream_scan_days: 2 + dream_max_units: 5 + +output: + dir: "benchmark/results/longmemeval" + log_dir: "logs" # log directory (relative to project root) + log_prefix: "longmemeval" # benchmark name used in log filenames + log_to_console: true + log_to_file: true diff --git a/benchmark/longmemeval/run.py b/benchmark/longmemeval/run.py new file mode 100644 index 00000000..f974b3a5 --- /dev/null +++ b/benchmark/longmemeval/run.py @@ -0,0 +1,763 @@ +"""LongMemEval evaluation runner for ReMe. + +Evaluates ReMe's long-term memory capability using the LongMemEval dataset. +Each item gets an isolated workspace; sessions are ingested in chronological order; +dream is triggered when sessions cross midnight (23:00); finally questions are +answered via an agentic (ReAct) approach and judged by an LLM. + +Usage: + python benchmark/longmemeval/run.py + python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml + python benchmark/longmemeval/run.py -q # quiet: only eval-level logs + python benchmark/longmemeval/run.py --log-level WARNING # reduce eval runner logs + python benchmark/longmemeval/run.py --reme-log-level WARNING # reduce reme internal logs + python benchmark/longmemeval/run.py --eval_only # query+judge only, reuse existing workspace +""" + +import json +import logging +import os +import re +import shutil +import time +import threading +from datetime import datetime +from pathlib import Path + +import yaml +from dotenv import load_dotenv + +# Load .env from project root +_PROJECT_ROOT = Path(__file__).parent.parent.parent +load_dotenv(_PROJECT_ROOT / ".env") + +# Workspace root for evaluation items — read from config.yaml (dataset.workspace_root) +_WORKSPACE_ROOT_DEFAULT = "benchmark/memory_workspaces/longmemeval-s" + +# --------------------------------------------------------------------------- +# Logging +# --------------------------------------------------------------------------- +_DEFAULT_LOG_FORMAT = "%(asctime)s | %(levelname)s | %(message)s" + +logging.basicConfig(level=logging.INFO, format=_DEFAULT_LOG_FORMAT) +logger = logging.getLogger("longmemeval") + +# Noisy library loggers silenced by default +_NOISY_LOGGERS = [ + "httpx", + "httpcore", + "openai", + "uvicorn", + "multipart", + "asyncio", + "watchfiles", + "filelock", +] + + +def setup_logging( + log_level: str, + reme_log_level: str, + log_dir: str | None = None, +): + """Configure logging for the eval runner and reme internals. + + Args: + log_level: Level for the eval runner logger (DEBUG/INFO/WARNING/ERROR). + reme_log_level: Level for reme's internal loguru logger. + log_dir: Per-run log directory (absolute path). None = no file logging. + """ + numeric = getattr(logging, log_level.upper(), logging.INFO) + # Eval runner logger + logging.getLogger().setLevel(numeric) + logger.setLevel(numeric) + + # Suppress noisy library loggers when above DEBUG + if numeric > logging.DEBUG: + for name in _NOISY_LOGGERS: + lib_logger = logging.getLogger(name) + lib_logger.setLevel(max(numeric, logging.WARNING)) + + # Add file handler for eval runner if log_dir is specified + if log_dir: + os.makedirs(log_dir, exist_ok=True) + log_filepath = os.path.join(log_dir, "runner.log") + file_handler = logging.FileHandler(log_filepath, encoding="utf-8") + file_handler.setLevel(numeric) + file_handler.setFormatter(logging.Formatter(_DEFAULT_LOG_FORMAT)) + logging.getLogger().addHandler(file_handler) + logger.info(f"Eval runner log file: {log_filepath}") + + # Reme internal logger (loguru) — will be applied per-worker via _configure_worker + os.environ["REME_LOG_LEVEL"] = reme_log_level.upper() + if log_dir: + os.environ["REME_LOG_DIR"] = log_dir + + +def _configure_worker( + log_level: str, + reme_log_level: str, + log_dir: str | None = None, +): + """Set up logging inside a multiprocessing worker process. + + Must be called at the top of each worker because child processes inherit + parent state but loguru sinks are NOT shared across fork/spawn. + """ + numeric = getattr(logging, log_level.upper(), logging.INFO) + logging.basicConfig(level=numeric, format=_DEFAULT_LOG_FORMAT, force=True) + logging.getLogger("longmemeval").setLevel(numeric) + if numeric > logging.DEBUG: + for name in _NOISY_LOGGERS: + logging.getLogger(name).setLevel(max(numeric, logging.WARNING)) + + # Add file handler for eval runner in worker process + if log_dir: + os.makedirs(log_dir, exist_ok=True) + pid = os.getpid() + log_filepath = os.path.join(log_dir, f"worker-{pid}.log") + file_handler = logging.FileHandler(log_filepath, encoding="utf-8") + file_handler.setLevel(numeric) + file_handler.setFormatter(logging.Formatter(_DEFAULT_LOG_FORMAT)) + logging.getLogger().addHandler(file_handler) + + # Re-initialize loguru for reme internals at the desired level + from reme.utils import get_logger + + reme_log_dir = log_dir or "logs" + get_logger(log_dir=reme_log_dir, level=reme_log_level.upper(), force_init=True) + + +# --------------------------------------------------------------------------- +# Config loading +# --------------------------------------------------------------------------- +def load_eval_config(config_path: str | None = None) -> dict: + """Load evaluation config yaml with env-var expansion.""" + if config_path is None: + config_path = str(Path(__file__).parent / "config.yaml") + with open(config_path, encoding="utf-8") as f: + raw = f.read() + + # Expand ${VAR} and ${VAR:-default} + def _expand(m): + expr = m.group(1) + if ":-" in expr: + key, default = expr.split(":-", 1) + return os.environ.get(key, default) + return os.environ.get(expr, "") + + raw = re.sub(r"\$\{([^}]+)\}", _expand, raw) + return yaml.safe_load(raw) + + +# --------------------------------------------------------------------------- +# Date utilities +# --------------------------------------------------------------------------- +def parse_haystack_date(date_str: str) -> datetime: + """Parse LongMemEval date format: '2023/05/20 (Sat) 02:21' -> datetime.""" + m = re.match(r"(\d{4}/\d{2}/\d{2})\s+\(\w+\)\s+(\d{2}:\d{2})", date_str) + if not m: + raise ValueError(f"Cannot parse haystack date: {date_str!r}") + return datetime.strptime(f"{m.group(1)} {m.group(2)}", "%Y/%m/%d %H:%M") + + +def to_iso(dt: datetime) -> str: + """Convert datetime to ISO-8601 string precise to seconds.""" + return dt.strftime("%Y-%m-%dT%H:%M:%S") + + +def should_trigger_dream(prev_dt: datetime, curr_dt: datetime, _trigger_hour: int = 23) -> bool: + """Check if the time gap between two sessions crosses trigger_hour (e.g. 23:00).""" + if prev_dt.date() == curr_dt.date(): + return False + # There's at least one midnight crossing; check if trigger_hour is between them + # Simple heuristic: if dates differ, dream should run for the previous day + return True + + +def sessions_sorted_by_time(item: dict) -> list[tuple[int, datetime, str, list[dict]]]: + """Return (original_index, parsed_datetime, session_id, messages) sorted by time.""" + entries = [] + for i, (date_str, sid, msgs) in enumerate( + zip(item["haystack_dates"], item["haystack_session_ids"], item["haystack_sessions"]), + ): + dt = parse_haystack_date(date_str) + entries.append((i, dt, sid, msgs)) + # Sort by time (ascending) + entries.sort(key=lambda x: x[1]) + return entries + + +# --------------------------------------------------------------------------- +# Message formatting +# --------------------------------------------------------------------------- +def format_messages_for_reme(messages: list[dict], session_dt: datetime) -> list[dict]: + """Convert LongMemEval messages to ReMe auto_memory format. + + Adds: name, created_at (ISO seconds). All messages in a session share the + same created_at (the session timestamp). + """ + formatted = [] + for msg in messages: + role = msg["role"] + formatted.append( + { + "name": role, + "role": role, + "content": msg["content"], + "created_at": to_iso(session_dt), + }, + ) + return formatted + + +# --------------------------------------------------------------------------- +# LLM-as-Judge (delegated to answer_judge_step via app.run_job) +# --------------------------------------------------------------------------- + + +async def judge_response_via_job( + app, + question: str, + ground_truth: str, + response: str, + question_type: str, +) -> dict: + """Use the answer_judge_step to evaluate a response against the golden answer.""" + judge_resp = await app.run_job( + "answer_judge", + query=question, + agent_answer=response, + golden_answer=ground_truth, + question_type=question_type, + ) + + verdict = (judge_resp.answer or "").strip().lower() + raw_answer = (judge_resp.metadata or {}).get("raw_answer_judgement", "") + + return { + "verdict": verdict, + "reason": raw_answer if verdict not in ("yes", "no") else "", + "metric": "binary", + "question_type": question_type, + } + + +# --------------------------------------------------------------------------- +# Main evaluation pipeline +# --------------------------------------------------------------------------- +async def evaluate_item(item: dict, eval_config: dict, item_index: int, eval_only: bool = False) -> dict: + """Evaluate a single LongMemEval item end-to-end. + + Args: + item: The dataset item containing question, answer, sessions, etc. + eval_config: The evaluation configuration dict. + item_index: The index of this item in the dataset. + eval_only: If True, skip ingestion (phases 1-3) and only run query+judge + using the existing workspace. Useful for re-evaluating different query + configurations without re-ingesting sessions. + """ + from reme import Application + from reme.config import resolve_app_config + + reme_cfg = eval_config["reme"] + dream_trigger_hour = reme_cfg.get("dream_trigger_hour", 23) + dream_scan_days = reme_cfg.get("dream_scan_days", 2) + dream_max_units = reme_cfg.get("dream_max_units", 5) + + # Sort sessions by time + sorted_sessions = sessions_sorted_by_time(item) + + # Filter out sessions that occur after question_date (if enabled) + filter_future = eval_config["evaluation"].get("filter_future_sessions", True) + if filter_future and item.get("question_date"): + question_dt = parse_haystack_date(item["question_date"]) + total_before_filter = len(sorted_sessions) + sorted_sessions = [(i, dt, sid, msgs) for i, dt, sid, msgs in sorted_sessions if dt <= question_dt] + if len(sorted_sessions) < total_before_filter: + logger.info( + f"[Item {item_index}] Filtered sessions: {total_before_filter} -> {len(sorted_sessions)} " + f"(removed {total_before_filter - len(sorted_sessions)} future sessions " + f"after question_date={item['question_date']})", + ) + + logger.info( + "[Item %s] question_id=%s type=%s sessions=%d%s", + item_index, + item["question_id"], + item["question_type"], + len(sorted_sessions), + " [eval_only]" if eval_only else "", + ) + + # Use fixed workspace directory (clean it for fresh evaluation) + workspace_root = _PROJECT_ROOT / eval_config["dataset"].get("workspace_root", _WORKSPACE_ROOT_DEFAULT) + item_dir = workspace_root / f"item_{item_index}" + workspace_dir = str(item_dir / ".reme") + if eval_only: + if not item_dir.exists() or not Path(workspace_dir).exists(): + raise FileNotFoundError( + f"[Item {item_index}] eval_only: workspace not found at {item_dir}. " + f"Run without --eval_only first to build the workspace.", + ) + else: + if item_dir.exists(): + shutil.rmtree(item_dir) + logger.info(f"[Item {item_index}] Cleaned existing workspace: {item_dir}") + else: + logger.info(f"[Item {item_index}] Workspace not found, creating: {item_dir}") + item_dir.mkdir(parents=True, exist_ok=True) + + # Pre-initialize ReMe's loguru logger with the correct log_dir + # (singleton — Application.__init__ will reuse this instance) + output_cfg = eval_config.get("output", {}) + if output_cfg.get("log_to_file", False): + reme_log_dir = os.environ.get("REME_LOG_DIR") + if reme_log_dir: + from reme.utils import get_logger + + get_logger( + log_dir=reme_log_dir, + level=os.environ.get("REME_LOG_LEVEL", "INFO"), + log_to_console=output_cfg.get("log_to_console", True), + log_to_file=True, + force_init=True, + ) + + cfg = resolve_app_config( + config=reme_cfg["config"], + workspace_dir=workspace_dir, + log_to_console=output_cfg.get("log_to_console", True), + log_to_file=output_cfg.get("log_to_file", False), + enable_logo=False, + ) + + app = Application(**cfg) + await app.start() + + try: + dream_dates_triggered = set() + dream_available = True # Set to False if auto_dream job is not found + + if not eval_only: + # ── Phase 1: Ingest sessions ────────────────────────────── + prev_dt = None + + for idx, (_, session_dt, session_id, messages) in enumerate(sorted_sessions): + # Check if dream should be triggered before this session + if ( + dream_available + and prev_dt is not None + and should_trigger_dream(prev_dt, session_dt, dream_trigger_hour) + ): + dream_date = prev_dt.strftime("%Y-%m-%d") + if dream_date not in dream_dates_triggered: + logger.info(f"[Item {item_index}] Triggering dream for date={dream_date}") + try: + dream_resp = await app.run_job( + "auto_dream", + date=dream_date, + scan_days=dream_scan_days, + max_units=dream_max_units, + ) + logger.info( + f"[Item {item_index}] Dream done: success={dream_resp.success} " + f"answer={dream_resp.answer[:100] if dream_resp.answer else ''}", + ) + except Exception as e: + if "not found" in str(e).lower(): + dream_available = False + logger.warning(f"[Item {item_index}] auto_dream job not found, skipping all dreams") + else: + logger.warning(f"[Item {item_index}] Dream failed for {dream_date}: {e}") + dream_dates_triggered.add(dream_date) + # Index update after dream to pick up new digest nodes + await app.run_job("index_update") + + # Format and ingest the session + formatted_msgs = format_messages_for_reme(messages, session_dt) + date_str = session_dt.strftime("%Y-%m-%d") + + logger.info( + f"[Item {item_index}] Ingesting session {idx+1}/{len(sorted_sessions)} " + f"id={session_id} date={date_str} msgs={len(formatted_msgs)}", + ) + resp = await app.run_job( + "auto_memory", + messages=formatted_msgs, + session_id=session_id, + date=date_str, + ) + if not resp.success: + logger.warning( + f"[Item {item_index}] auto_memory failed for session {session_id}: {resp.answer}", + ) + + # Manual index update after each session + await app.run_job("index_update") + + prev_dt = session_dt + + # ── Phase 2: Final dream for the last day ───────────────── + if dream_available and prev_dt is not None: + last_dream_date = prev_dt.strftime("%Y-%m-%d") + if last_dream_date not in dream_dates_triggered: + logger.info(f"[Item {item_index}] Final dream for date={last_dream_date}") + try: + await app.run_job( + "auto_dream", + date=last_dream_date, + scan_days=dream_scan_days, + max_units=dream_max_units, + ) + except Exception as e: + if "not found" in str(e).lower(): + dream_available = False + logger.warning(f"[Item {item_index}] auto_dream job not found, skipping all dreams") + else: + logger.warning(f"[Item {item_index}] Final dream failed: {e}") + dream_dates_triggered.add(last_dream_date) + # Index update after final dream + await app.run_job("index_update") + + # ── Phase 3: Digest update ──────────────────────────────── + await app.run_job("digest_update") + + # ── Phase 4: Ask question via agentic_answer job (ReAct agent) ── + question = item["question"] + question_date_raw = item.get("question_date", "") + question_dt = parse_haystack_date(question_date_raw) if question_date_raw else None + query_time = to_iso(question_dt) if question_dt else "" + logger.info( + f"[Item {item_index}] Asking (agentic): {question[:80]}... query_time={query_time}", + ) + + query_resp = await app.run_job( + "agentic_answer", + query=question, + query_time=query_time, + ) + agentic_response = (query_resp.answer or "").strip() + if not agentic_response: + agentic_response = "(no answer generated)" + + logger.info(f"[Item {item_index}] Agentic response: {agentic_response[:200]}...") + + # ── Phase 5: Judge agentic response (via answer_judge_step) ────────── + logger.info(f"[Item {item_index}] Judging agentic (binary, type={item['question_type']})...") + agentic_judgment = await judge_response_via_job( + app=app, + question=question, + ground_truth=item["answer"], + response=agentic_response, + question_type=item["question_type"], + ) + logger.info(f"[Item {item_index}] agentic binary result: {agentic_judgment}") + + finally: + await app.close() + + return { + "question_id": item["question_id"], + "question_type": item["question_type"], + "question": question, + "ground_truth": item["answer"], + "agentic_response": agentic_response, + "agentic_judgment": agentic_judgment, + "sessions_ingested": len(sorted_sessions), + "dreams_triggered": len(dream_dates_triggered), + } + + +# --------------------------------------------------------------------------- +# Worker: runs a single item in its own process with its own event loop +# --------------------------------------------------------------------------- +def _evaluate_item_worker(task_input: tuple) -> dict: + """Worker function for multiprocessing. Each process gets its own event loop.""" + item, eval_config, item_index, log_level, reme_log_level, eval_only, log_dir = task_input + import asyncio # pylint: disable=import-outside-toplevel + + _configure_worker(log_level, reme_log_level, log_dir=log_dir) + + # Permanently suppress "Task exception was never retrieved" / + # "Event loop is closed" noise from httpx AsyncClient GC cleanup. + # These fire AFTER asyncio.run() closes the loop, during Python's + # garbage collection of httpx connection-pool tasks — harmless. + logging.getLogger("asyncio").setLevel(logging.CRITICAL) + + return asyncio.run(evaluate_item(item, eval_config, item_index, eval_only=eval_only)) + + +def _indexed_worker(indexed_input: tuple) -> tuple: + """Module-level wrapper for imap_unordered with index tracking.""" + idx, task_input = indexed_input + return idx, _evaluate_item_worker(task_input) + + +def _resolve_num_workers(configured: int) -> int: + """Resolve num_workers: 0=auto (cpu_count-2, min 1), 1=sequential, >1=parallel.""" + if configured == 0: + return max(1, (os.cpu_count() or 4) - 2) + return max(1, configured) + + +# --------------------------------------------------------------------------- +# Entry point +# --------------------------------------------------------------------------- +def main( + config_path: str | None = None, + log_level: str = "INFO", + reme_log_level: str = "INFO", + eval_only: bool = False, +): + """Run the LongMemEval evaluation pipeline. + + Args: + config_path: Path to the YAML config file. + log_level: Log level for the eval runner. + reme_log_level: Log level for reme internal logs. + eval_only: If True, skip ingestion and only run query+judge using + existing workspaces. + """ + from multiprocessing import Pool # pylint: disable=import-outside-toplevel + + # Load config BEFORE logging setup so log_dir is available + eval_config = load_eval_config(config_path) + + # Resolve per-run log directory from config + output_cfg = eval_config.get("output", {}) + log_dir_abs = None + if output_cfg.get("log_to_file", False): + log_dir_raw = output_cfg.get("log_dir", "logs") + log_prefix = output_cfg.get("log_prefix", "longmemeval") + run_ts = datetime.now().strftime("%Y-%m-%d_%H-%M-%S") + log_dir_abs = str(_PROJECT_ROOT / log_dir_raw / f"{log_prefix}_{run_ts}") + + setup_logging(log_level, reme_log_level, log_dir=log_dir_abs) + dataset_cfg = eval_config["dataset"] + + # Load dataset + dataset_path = _PROJECT_ROOT / dataset_cfg["path"] + logger.info(f"Loading dataset from {dataset_path}") + with open(dataset_path, encoding="utf-8") as f: + data = json.load(f) + + start = dataset_cfg.get("start_index", 0) + num_items = dataset_cfg.get("num_items", 0) + if num_items > 0: + raw_items = data[start : start + num_items] + else: + raw_items = data[start:] + + # Build item list + items_with_idx = [(start + i, item) for i, item in enumerate(raw_items)] + + # Filter by question_type if specified + question_types = dataset_cfg.get("question_types") or [] + if question_types: + before_filter = len(items_with_idx) + items_with_idx = [(idx, item) for idx, item in items_with_idx if item.get("question_type") in question_types] + logger.info( + f"Filtered by question_types={question_types}: {before_filter} -> {len(items_with_idx)} items", + ) + + logger.info( + "Evaluating %d item(s) starting from index %d%s", + len(items_with_idx), + start, + " [eval_only: query+judge only]" if eval_only else "", + ) + + # Resolve parallelism + num_workers = _resolve_num_workers(eval_config["evaluation"].get("num_workers", 1)) + logger.info(f"Using {num_workers} worker(s)") + + # Create output directory + output_dir = _PROJECT_ROOT / output_cfg.get("dir", "benchmark/results/longmemeval") + output_dir.mkdir(parents=True, exist_ok=True) + + # Create workspace root directory + workspace_root = _PROJECT_ROOT / dataset_cfg.get("workspace_root", _WORKSPACE_ROOT_DEFAULT) + workspace_root.mkdir(parents=True, exist_ok=True) + + # Pre-check: verify all workspaces exist in eval_only mode + if eval_only: + missing_items = [] + for orig_idx, _ in items_with_idx: + item_dir = workspace_root / f"item_{orig_idx}" + if not item_dir.exists() or not (item_dir / ".reme").exists(): + missing_items.append(orig_idx) + if missing_items: + preview = missing_items[:10] + suffix = "..." if len(missing_items) > 10 else "" + raise FileNotFoundError( + f"eval_only: {len(missing_items)} workspace(s) not found under {workspace_root}. " + f"Missing item indices: {preview}{suffix}. " + f"Run without --eval_only first to build the workspaces.", + ) + + # Build task args — include log levels, eval_only flag, and log paths (use original index for workspace lookup) + task_args = [ + (item, eval_config, orig_idx, log_level, reme_log_level, eval_only, log_dir_abs) + for orig_idx, item in items_with_idx + ] + + # Progress tracking (force print regardless of log level, every 10 minutes) + total_items = len(task_args) + completed_count = [0] # use list for mutability in closure + start_time = time.time() + progress_lock = threading.Lock() + + def _print_progress(prefix: str = "PROGRESS"): + elapsed = time.time() - start_time + elapsed_min = elapsed / 60 + done = completed_count[0] + pct = 100.0 * done / total_items if total_items else 0 + eta_str = "N/A" + if done > 0: + eta_sec = elapsed / done * (total_items - done) + eta_str = f"{eta_sec/60:.1f}min" + print( + f"[{prefix}] {datetime.now().strftime('%Y-%m-%d %H:%M:%S')} | " + f"{done}/{total_items} ({pct:.1f}%) completed | " + f"elapsed={elapsed_min:.1f}min | ETA={eta_str}", + flush=True, + ) + + def _progress_timer(): + """Background thread: print progress every 10 minutes.""" + while not _timer_stop.is_set(): + _timer_stop.wait(600) # 10 minutes + if not _timer_stop.is_set(): + with progress_lock: + _print_progress() + + _timer_stop = threading.Event() + timer_thread = threading.Thread(target=_progress_timer, daemon=True) + timer_thread.start() + + # Run evaluation + if num_workers == 1: + # Sequential mode + results = [] + for task_input in task_args: + result = _evaluate_item_worker(task_input) + results.append(result) + with progress_lock: + completed_count[0] += 1 + else: + # Parallel mode — use imap_unordered for progress tracking + results = [None] * total_items + indexed_args = list(enumerate(task_args)) + + with Pool(processes=num_workers) as pool: + for idx, result in pool.imap_unordered(_indexed_worker, indexed_args): + results[idx] = result + with progress_lock: + completed_count[0] += 1 + + # Stop progress timer + _timer_stop.set() + timer_thread.join(timeout=2) + + # Save results + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + output_file = output_dir / f"results_{timestamp}.json" + with open(output_file, "w", encoding="utf-8") as f: + json.dump(results, f, ensure_ascii=False, indent=2) + logger.info(f"Results saved to {output_file}") + + # Final progress + _print_progress("FINAL") + + _print_summary(results, start_time) + + +# --------------------------------------------------------------------------- +# Summary printing +# --------------------------------------------------------------------------- +def _print_summary(results: list[dict], start_time: float) -> None: + """Print per-item verdicts and per-type accuracy.""" + print("\n" + "=" * 60) + print("EVALUATION RESULTS") + print("=" * 60) + + def _accumulate(judgment_key): + correct = 0 + stats: dict = {} # {question_type: {correct: int, total: int}} + for r in results: + qtype = r["question_type"] + verdict = r.get(judgment_key, {}).get("verdict", "N/A") + if qtype not in stats: + stats[qtype] = {"correct": 0, "total": 0} + stats[qtype]["total"] += 1 + if verdict == "yes": + correct += 1 + stats[qtype]["correct"] += 1 + return correct, stats + + agentic_correct, agentic_type_stats = _accumulate("agentic_judgment") + + total = len(results) + + # Per-item verdict rows + for r in results: + a_verdict = r.get("agentic_judgment", {}).get("verdict", "N/A") + print(f" [{r['question_id']}] type={r['question_type']} agentic={a_verdict}") + + print("\n" + "-" * 60) + print(f" Items: {total}") + + # Agentic stats + print("\n ── Agentic (ReAct) ──") + print(f" Overall accuracy: {agentic_correct}/{total} ({100*agentic_correct/total:.1f}%)") + print(" Per-type accuracy:") + for qtype, stats in sorted(agentic_type_stats.items()): + acc = 100 * stats["correct"] / stats["total"] if stats["total"] else 0 + print(f" {qtype}: {stats['correct']}/{stats['total']} ({acc:.1f}%)") + + print("=" * 60) + total_elapsed = time.time() - start_time + print(f"\n Total time: {total_elapsed/60:.1f} min") + print("\n" + "=" * 60) + print(" [DONE] EVALUATION COMPLETED SUCCESSFULLY") + print("=" * 60 + "\n") + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser(description="LongMemEval evaluation runner") + parser.add_argument("--config", type=str, default=None, help="Path to config.yaml") + parser.add_argument( + "--log-level", + type=str, + default="INFO", + choices=["DEBUG", "INFO", "WARNING", "ERROR"], + help="Log level for the eval runner (default: INFO)", + ) + parser.add_argument( + "--reme-log-level", + type=str, + default="INFO", + choices=["DEBUG", "INFO", "WARNING", "ERROR"], + help="Log level for reme internal logs — loguru (default: INFO)", + ) + parser.add_argument( + "-q", + "--quiet", + action="store_true", + help="Shortcut for --log-level WARNING --reme-log-level WARNING", + ) + parser.add_argument( + "--eval_only", + action="store_true", + help="Skip ingestion (phases 1-3). Reuse existing workspaces and only run query+judge.", + ) + args = parser.parse_args() + + if args.quiet: + args.log_level = "WARNING" + args.reme_log_level = "WARNING" + + main(args.config, args.log_level, args.reme_log_level, eval_only=args.eval_only) diff --git a/benchmark/longmemeval/run_agentic_answer.py b/benchmark/longmemeval/run_agentic_answer.py deleted file mode 100644 index f5bc1631..00000000 --- a/benchmark/longmemeval/run_agentic_answer.py +++ /dev/null @@ -1,343 +0,0 @@ -#!/usr/bin/env python3 -"""Drive the LongMemEval memory pipeline across all samples. - -For every workspace under ``datasets/longmemeval/`` this launches one or more -``reme start config=jinli_lme job=`` runs with ``LME_WORKSPACE_DIR`` pointed -at that sample. The pipeline jobs, in order, are: - - 1. auto_memory — distil every raw session into a daily note (``daily/*.md``) - 2. update_index — clear the store and rebuild the index over ``daily/*.md`` - 3. agentic_answer — read ``query.json`` and answer it, writing ``mem_answer.json`` - 4. llm_judge — judge ``mem_answer.json`` against ``answer.json`` - -Pick one with ``--job``, or ``--job all`` to run the full pipeline *serially per sample*. -Runs are capped at ``--concurrency`` (default 1 for ``--job auto_memory``, otherwise -3) samples at once and each launch is staggered by ``--stagger`` seconds so they -do not all hit the LLM API at once. - -By default every selected job is rerun for every sample — each job's own clear -step (configured in jinli_lme.yaml) wipes stale output first, so a run is always -a clean rebuild. Pass ``--resume`` to instead skip samples whose output already -exists (``daily/`` for auto_memory, ``metadata/embedding_store/`` for -update_index, ``mem_answer.json`` for agentic_answer, ``mem_answer.json`` with -``llm_judge.judgement`` for llm_judge) and continue an interrupted batch. Each -sample's stdout/stderr goes to ``logs/agentic_answer//.log``. - -After an agentic_answer run finishes, the driver aggregates every sample's query, -golden answer, predicted answer, LLM judgement and a best-effort tool-call trail -into one big JSON at ``logs/agentic_answer/aggregate.json``. - -Examples: - python benchmark/longmemeval/run_agentic_answer.py # agentic_answer, all 500, conc 3 - python benchmark/longmemeval/run_agentic_answer.py --job all # full pipeline serially per sample - python benchmark/longmemeval/run_agentic_answer.py --job auto_memory # just step 1 - python benchmark/longmemeval/run_agentic_answer.py --job llm_judge # just judge existing answers - python benchmark/longmemeval/run_agentic_answer.py --limit 5 --dry-run # list what would run - python benchmark/longmemeval/run_agentic_answer.py --start 187 # samples 187..499 - python benchmark/longmemeval/run_agentic_answer.py --start 187 --end 499 # samples 187..499 - python benchmark/longmemeval/run_agentic_answer.py --job all --resume # continue an interrupted batch -""" - -import argparse -import asyncio -import json -import os -import re -import time -from pathlib import Path - -REPO = Path(__file__).resolve().parents[2] -DATA = REPO / "datasets" / "longmemeval" -LOGDIR = REPO / "logs" / "agentic_answer" -AGGREGATE = LOGDIR / "aggregate.json" - -# Pipeline jobs in execution order. -JOB_ORDER = ["auto_memory", "update_index", "agentic_answer", "llm_judge"] - - -def parse_args() -> argparse.Namespace: - """Parse command-line arguments.""" - p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - p.add_argument( - "--job", - choices=[*JOB_ORDER, "all"], - default="agentic_answer", - help="which job to run per sample; 'all' runs the full pipeline serially (default: agentic_answer)", - ) - p.add_argument("--concurrency", type=int, default=1, help="max samples running at once (default 3)") - p.add_argument("--stagger", type=float, default=1.0, help="seconds between consecutive launches (default 1)") - p.add_argument("--start", type=int, default=0, help="first numeric sample id to process, inclusive (default 0)") - p.add_argument( - "--end", - type=int, - default=0, - help="last numeric sample id to process, inclusive (0 = no upper bound)", - ) - p.add_argument("--limit", type=int, default=0, help="only process the first N samples (0 = all)") - p.add_argument( - "--resume", - action="store_true", - help="skip a sample when the job's output already exists (resume an interrupted run); " - "by default every selected job is rerun so the config's clear step rebuilds cleanly", - ) - p.add_argument("--dry-run", action="store_true", help="list what would run, launch nothing") - p.add_argument("--no-aggregate", action="store_true", help="skip writing aggregate.json after answer/judge jobs") - return p.parse_args() - - -def selected_jobs(job: str) -> list[str]: - """Expand the --job choice into an ordered list of jobs.""" - return list(JOB_ORDER) if job == "all" else [job] - - -def sample_ids() -> list[str]: - """List all sample IDs (numeric workspace dirs), numerically sorted.""" - ids = [p.name for p in DATA.iterdir() if p.is_dir() and p.name.isdigit()] - return sorted(ids, key=int) - - -def job_done(idx: str, job: str) -> bool: - """Return True when ``job``'s expected output already exists for sample ``idx``.""" - ws = DATA / idx - if job == "auto_memory": - daily = ws / "daily" - return daily.is_dir() and any(daily.rglob("*.md")) - if job == "update_index": - store = ws / "metadata" / "embedding_store" - return store.is_dir() and any(store.iterdir()) - if job == "agentic_answer": - return (ws / "mem_answer.json").exists() - if job == "llm_judge": - judge = _load_json(ws / "mem_answer.json").get("llm_judge") - return isinstance(judge, dict) and bool(str(judge.get("judgement") or "").strip()) - raise ValueError(f"unknown job: {job}") - - -async def run_job(idx: str, job: str, counters: dict) -> bool: - """Run a single job for a single sample. Returns True on success.""" - log = LOGDIR / job / f"{idx}.log" - log.parent.mkdir(parents=True, exist_ok=True) - env = dict(os.environ, LME_WORKSPACE_DIR=f"datasets/longmemeval/{idx}") - started = time.strftime("%H:%M:%S") - print(f"[start {started}] {idx}/{job}", flush=True) - with log.open("w", encoding="utf-8") as f: - proc = await asyncio.create_subprocess_exec( - "reme", - "start", - "config=jinli_lme", - f"job={job}", - cwd=str(REPO), - env=env, - stdout=f, - stderr=asyncio.subprocess.STDOUT, - ) - rc = await proc.wait() - ok = rc == 0 and job_done(idx, job) - counters["done" if ok else "fail"] += 1 - tag = "done" if ok else "fail" - print(f"[{tag}] {idx}/{job} rc={rc} ({counters['done']} done / {counters['fail']} fail)", flush=True) - return ok - - -async def run_one(idx: str, jobs: list[str], sem: asyncio.Semaphore, resume: bool, counters: dict) -> None: - """Run the selected jobs for one sample, serially. - - By default every selected job is rerun (the job's own clear step wipes stale - output first). With ``resume`` a job is skipped when its output already - exists, so an interrupted batch can continue without redoing finished work. - """ - async with sem: - for job in jobs: - if resume and job_done(idx, job): - counters["skip"] += 1 - print(f"[skip] {idx}/{job} (output exists)", flush=True) - continue - ok = await run_job(idx, job, counters) - if not ok: - # Later jobs depend on earlier ones; don't waste a run on a broken workspace. - print(f"[abort] {idx}: {job} failed, skipping remaining jobs", flush=True) - break - - -# --------------------------------------------------------------------------- # -# Aggregation of agentic_answer results into one big JSON. -# --------------------------------------------------------------------------- # - -# Match ``session_id=abc123`` headers and ``"...session_id": "abc123"`` fields in -# tool-result text, so we can list which sessions each search actually surfaced. -_SID_RE = re.compile(r'session_id["\s:=]+"?([A-Za-z0-9_\-]+)') - - -def _load_json(path: Path) -> dict: - """Load a JSON object, returning {} on any error.""" - try: - with path.open(encoding="utf-8") as f: - data = json.load(f) - return data if isinstance(data, dict) else {} - except (OSError, json.JSONDecodeError): - return {} - - -def parse_tool_calls(idx: str, session_id: str) -> list[dict]: - """Best-effort: parse the agent trajectory into an ordered tool-call summary. - - Reads ``mem_session/agentscope/.jsonl`` — the trajectory the - agentic_answer run dumped — and pairs every ``tool_call`` (name + parsed - args) with the ``session_id`` hits found in its ``tool_result``. Returns an - empty list if the file is missing or unreadable (never raises). - """ - if not session_id: - return [] - path = DATA / idx / "mem_session" / "agentscope" / f"{session_id}.jsonl" - if not path.exists(): - return [] - - calls: dict[str, dict] = {} - order: list[str] = [] - try: - for line in path.read_text(encoding="utf-8").splitlines(): - line = line.strip() - if not line: - continue - try: - msg = json.loads(line) - except json.JSONDecodeError: - continue - for c in msg.get("content") or []: - if not isinstance(c, dict): - continue - cid = c.get("id") - if c.get("type") == "tool_call" and cid: - try: - args = json.loads(c.get("input") or "{}") - except (json.JSONDecodeError, TypeError): - args = c.get("input") - calls[cid] = {"name": c.get("name"), "args": args, "hit_session_ids": []} - order.append(cid) - elif c.get("type") == "tool_result" and cid in calls: - text = "" - for o in c.get("output") or []: - if isinstance(o, dict) and isinstance(o.get("text"), str): - text += o["text"] - hits = list(dict.fromkeys(_SID_RE.findall(text))) - calls[cid]["hit_session_ids"] = hits - except OSError: - return [] - - return [{"iter": i + 1, **calls[cid]} for i, cid in enumerate(order)] - - -def build_record(idx: str) -> dict: - """Assemble one sample's aggregate record from its on-disk artifacts.""" - ws = DATA / idx - query = _load_json(ws / "query.json") - golden = _load_json(ws / "answer.json") - mem = _load_json(ws / "mem_answer.json") - - pred = str(mem.get("answer") or "").strip() - session_id = str(mem.get("session_id") or "") - llm_judge = mem.get("llm_judge") if isinstance(mem.get("llm_judge"), dict) else {} - tool_calls = parse_tool_calls(idx, session_id) if mem else [] - - if not mem: - status = "missing" - elif not pred: - status = "empty" - elif "not provided" in pred.lower(): - status = "not_provided" - else: - status = "answered" - - return { - "idx": idx, - "question_id": query.get("question_id"), - "question_type": query.get("question_type"), - "question": query.get("question"), - "question_date": query.get("question_date"), - "golden_answer": golden.get("answer"), - "golden_answer_session_ids": golden.get("answer_session_ids"), - "pred_answer": pred, - "session_id": session_id, - "status": status, - "llm_judge": llm_judge.get("judgement"), - "llm_judge_raw": llm_judge.get("raw_judgement"), - "num_tool_calls": len(tool_calls), - "tool_calls": tool_calls, - } - - -def write_aggregate(ids: list[str]) -> None: - """Aggregate every sample's agentic_answer artifacts into one big JSON.""" - records = [build_record(idx) for idx in ids] - finished = [r for r in records if r["status"] != "missing"] - by_status: dict[str, int] = {} - by_llm_judge: dict[str, int] = {} - for r in records: - by_status[r["status"]] = by_status.get(r["status"], 0) + 1 - judgement = r.get("llm_judge") or "missing" - by_llm_judge[judgement] = by_llm_judge.get(judgement, 0) + 1 - - payload = { - "generated_at": time.strftime("%Y-%m-%d %H:%M:%S"), - "total": len(records), - "finished": len(finished), - "by_status": by_status, - "by_llm_judge": by_llm_judge, - "samples": records, - } - AGGREGATE.parent.mkdir(parents=True, exist_ok=True) - AGGREGATE.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8") - print(f"[aggregate] wrote {len(records)} samples ({len(finished)} finished) -> {AGGREGATE}", flush=True) - - -async def main() -> int: - """Run the driver.""" - args = parse_args() - LOGDIR.mkdir(parents=True, exist_ok=True) - jobs = selected_jobs(args.job) - - ids = sample_ids() - if args.end and args.end < args.start: - raise ValueError(f"--end ({args.end}) must be >= --start ({args.start})") - ids = [i for i in ids if int(i) >= args.start and (not args.end or int(i) <= args.end)] - if args.limit: - ids = ids[: args.limit] - - # Without --resume every job reruns; with --resume, jobs whose output exists are skipped. - def todo_jobs(i: str) -> list[str]: - return [j for j in jobs if not (args.resume and job_done(i, j))] - - pending = [i for i in ids if todo_jobs(i)] - print( - f"jobs={jobs} resume={args.resume} samples total={len(ids)} pending={len(pending)} " - f"concurrency={args.concurrency} stagger={args.stagger}s", - flush=True, - ) - - if args.dry_run: - for i in pending: - print(f"[would-run] {i}: {todo_jobs(i)}") - return 0 - - sem = asyncio.Semaphore(args.concurrency) - counters = {"done": 0, "fail": 0, "skip": 0} - tasks: list[asyncio.Task] = [] - for n, idx in enumerate(ids): - if n and args.stagger > 0: - await asyncio.sleep(args.stagger) # stagger each launch relative to the previous - tasks.append(asyncio.create_task(run_one(idx, jobs, sem, args.resume, counters))) - - await asyncio.gather(*tasks, return_exceptions=True) - print( - f"ALL FINISHED done={counters['done']} fail={counters['fail']} skip={counters['skip']}", - flush=True, - ) - - if any(j in jobs for j in ("agentic_answer", "llm_judge")) and not args.no_aggregate: - write_aggregate(ids) - - return 0 if counters["fail"] == 0 else 1 - - -if __name__ == "__main__": - raise SystemExit(asyncio.run(main())) diff --git a/benchmark/longmemeval/run_final_answer_review.py b/benchmark/longmemeval/run_final_answer_review.py deleted file mode 100644 index 2c8491ad..00000000 --- a/benchmark/longmemeval/run_final_answer_review.py +++ /dev/null @@ -1,469 +0,0 @@ -#!/usr/bin/env python3 -"""Review every LongMemEval golden answer with the configured Claude Code job. - -Every numeric ``datasets/longmemeval/`` workspace is processed sequentially. -The reference JSONL files are merged by ``question_id`` and supplied only when -they contain an alternative answer for that sample: - - reme start config=jinli_lme job=final_answer_review - -The job returns a plain four-field JSON object with ``reason``, -``golden_answer_correct``, ``answer``, and ``is_session_time_wrong``. After -every new success, this driver atomically rewrites the complete accumulated -output JSONL so an interrupted run can safely resume. - -Examples: - python benchmark/longmemeval/run_final_answer_review.py - python benchmark/longmemeval/run_final_answer_review.py --exclude-reference-question-ids - python benchmark/longmemeval/run_final_answer_review.py --only-reference-question-ids --rerun-selected - python benchmark/longmemeval/run_final_answer_review.py --concurrency 2 --submit-interval-seconds 6 - python benchmark/longmemeval/run_final_answer_review.py --question-id e47becba - python benchmark/longmemeval/run_final_answer_review.py --reference path/to/results.jsonl - python benchmark/longmemeval/run_final_answer_review.py --limit 3 - python benchmark/longmemeval/run_final_answer_review.py --no-resume - python benchmark/longmemeval/run_final_answer_review.py --dry-run -""" - -import argparse -import concurrent.futures -import json -import os -import subprocess -import sys -import tempfile -import time -from pathlib import Path -from typing import Any - -REPO = Path(__file__).resolve().parents[2] -DATA = REPO / "datasets" / "longmemeval" -DEFAULT_REFERENCES = ( - REPO / "benchmark" / "longmemeval" / "golden_check_list_false.jsonl", - REPO / "benchmark" / "longmemeval" / "merge_confirm_jinli_false.jsonl", -) -DEFAULT_OUTPUT = REPO / "benchmark" / "longmemeval" / "final_answer_review.jsonl" -DEFAULT_LOG_DIR = REPO / "logs" / "final_answer_review" -REFERENCE_PATHS_ENV = "LME_FINAL_ANSWER_REFERENCE_PATHS" -MAX_CONCURRENCY = 3 -MIN_SUBMIT_INTERVAL_SECONDS = 5.0 -DEFAULT_SUBMIT_INTERVAL_SECONDS = 5.1 - - -def parse_args() -> argparse.Namespace: - """Parse command-line arguments.""" - parser = argparse.ArgumentParser( - description=__doc__, - formatter_class=argparse.RawDescriptionHelpFormatter, - ) - parser.add_argument( - "--question-id", - dest="question_ids", - action="append", - help="process only this dataset question ID; repeat for multiple IDs (default: all)", - ) - reference_selection = parser.add_mutually_exclusive_group() - reference_selection.add_argument( - "--exclude-reference-question-ids", - action="store_true", - help="skip question IDs found in the selected reference-answer JSONL files", - ) - reference_selection.add_argument( - "--only-reference-question-ids", - action="store_true", - help="process only question IDs found in the selected reference-answer JSONL files", - ) - parser.add_argument( - "--reference", - dest="references", - action="append", - type=Path, - help="reference-answer JSONL; repeat for multiple files (default: built-in disputed results)", - ) - parser.add_argument( - "--output", - type=Path, - default=DEFAULT_OUTPUT, - help=f"output JSONL (default: {DEFAULT_OUTPUT})", - ) - parser.add_argument( - "--log-dir", - type=Path, - default=DEFAULT_LOG_DIR, - help="directory for per-question logs", - ) - parser.add_argument( - "--concurrency", - type=int, - default=MAX_CONCURRENCY, - help=f"maximum concurrent jobs, from 1 to {MAX_CONCURRENCY} (default: {MAX_CONCURRENCY})", - ) - parser.add_argument( - "--submit-interval-seconds", - type=float, - default=DEFAULT_SUBMIT_INTERVAL_SECONDS, - help=f"minimum time between job submissions; must be > {MIN_SUBMIT_INTERVAL_SECONDS:g} " - f"(default: {DEFAULT_SUBMIT_INTERVAL_SECONDS:g})", - ) - parser.add_argument( - "--limit", - type=int, - default=0, - help="process only the first N pending questions (0 = all)", - ) - resume_mode = parser.add_mutually_exclusive_group() - resume_mode.add_argument( - "--no-resume", - action="store_true", - help="ignore existing output and rerun every selected question", - ) - resume_mode.add_argument( - "--rerun-selected", - action="store_true", - help="rerun every selected question while preserving existing results until replacements finish", - ) - parser.add_argument( - "--dry-run", - action="store_true", - help="show the selected cases without invoking ReMe", - ) - return parser.parse_args() - - -def _read_jsonl(path: Path) -> list[dict[str, Any]]: - """Read a JSONL file and reject malformed or non-object rows.""" - rows: list[dict[str, Any]] = [] - try: - with path.open(encoding="utf-8") as file: - for line_number, line in enumerate(file, start=1): - if not line.strip(): - continue - try: - row = json.loads(line) - except json.JSONDecodeError as exc: - raise ValueError(f"Invalid JSON at {path}:{line_number}") from exc - if not isinstance(row, dict): - raise ValueError(f"Expected a JSON object at {path}:{line_number}") - rows.append(row) - except OSError as exc: - raise FileNotFoundError(f"Cannot read JSONL file: {path}") from exc - return rows - - -def merge_references(paths: list[Path]) -> dict[str, list[dict[str, Any]]]: - """Merge reference rows by question ID, preserving file and row order.""" - merged: dict[str, list[dict[str, Any]]] = {} - seen_sources: set[tuple[str, str]] = set() - for path in paths: - for row in _read_jsonl(path): - question_id = str(row.get("question_id") or "").strip() - if not question_id: - raise ValueError(f"Reference row in {path} has no question_id") - source_key = (question_id, str(path.resolve())) - if source_key in seen_sources: - raise ValueError(f"Duplicate question_id={question_id!r} within {path}") - seen_sources.add(source_key) - merged.setdefault(question_id, []).append({"source": path.name, **row}) - if not merged: - raise ValueError("No reference answers found") - return merged - - -def workspace_map() -> dict[str, Path]: - """Map every dataset question ID to its numeric sample workspace.""" - mapping: dict[str, Path] = {} - for workspace in sorted( - (path for path in DATA.iterdir() if path.is_dir() and path.name.isdigit()), - key=lambda p: int(p.name), - ): - query_path = workspace / "query.json" - if not query_path.is_file(): - continue - try: - with query_path.open(encoding="utf-8") as file: - query = json.load(file) - except (OSError, json.JSONDecodeError) as exc: - raise ValueError(f"Cannot parse {query_path}") from exc - if not isinstance(query, dict): - raise ValueError(f"Expected a JSON object in {query_path}") - question_id = str(query.get("question_id") or "").strip() - if not question_id: - raise ValueError(f"Missing question_id in {query_path}") - if question_id in mapping: - raise ValueError( - f"Duplicate dataset question_id={question_id!r}: {mapping[question_id]} and {workspace}", - ) - mapping[question_id] = workspace - return mapping - - -def select_question_ids( - mapping: dict[str, Path], - requested: list[str] | None, - excluded: set[str] | None = None, -) -> list[str]: - """Return all dataset IDs or validate an explicitly requested subset.""" - excluded = excluded or set() - if not requested: - return [question_id for question_id in mapping if question_id not in excluded] - selected: list[str] = [] - seen: set[str] = set() - for raw_question_id in requested: - question_id = raw_question_id.strip() - if not question_id: - raise ValueError("--question-id must not be empty") - if question_id in seen: - raise ValueError(f"Duplicate --question-id: {question_id}") - if question_id not in mapping: - raise ValueError(f"No dataset workspace for question ID: {question_id}") - if question_id not in excluded: - selected.append(question_id) - seen.add(question_id) - return selected - - -def _validate_result(value: Any, *, source: str) -> dict[str, Any]: - """Validate the final four-field answer contract.""" - expected_keys = {"reason", "golden_answer_correct", "answer", "is_session_time_wrong"} - if not isinstance(value, dict) or set(value) != expected_keys: - raise ValueError( - f"{source} must contain exactly 'reason', 'golden_answer_correct', 'answer', " - "and 'is_session_time_wrong'", - ) - if not isinstance(value["reason"], str) or not value["reason"].strip(): - raise ValueError(f"{source} has an invalid reason") - if not isinstance(value["golden_answer_correct"], bool): - raise ValueError(f"{source} has an invalid golden_answer_correct") - if not isinstance(value["answer"], str): - raise ValueError(f"{source} has an invalid answer") - answer = value["answer"].strip() - if value["golden_answer_correct"] and answer: - raise ValueError(f"{source} answer must be empty when golden_answer_correct is true") - if not value["golden_answer_correct"] and not answer: - raise ValueError(f"{source} answer must be non-empty when golden_answer_correct is false") - if not isinstance(value["is_session_time_wrong"], bool): - raise ValueError(f"{source} has an invalid is_session_time_wrong") - return { - "reason": value["reason"].strip(), - "golden_answer_correct": value["golden_answer_correct"], - "answer": answer, - "is_session_time_wrong": False, - } - - -def load_existing(path: Path) -> dict[str, dict[str, Any]]: - """Load resumable output, rejecting duplicate or malformed rows.""" - if not path.exists(): - return {} - results: dict[str, dict[str, Any]] = {} - for row in _read_jsonl(path): - question_id = str(row.get("question_id") or "").strip() - if not question_id: - raise ValueError(f"Existing output row in {path} has no question_id") - if question_id in results: - raise ValueError( - f"Duplicate question_id={question_id!r} in existing output {path}", - ) - results[question_id] = _validate_result( - {key: value for key, value in row.items() if key != "question_id"}, - source=f"existing result for {question_id}", - ) - return results - - -def atomic_write_results( - path: Path, - order: list[str], - results: dict[str, dict[str, Any]], -) -> None: - """Atomically rewrite all accumulated rows in stable merged-input order.""" - path.parent.mkdir(parents=True, exist_ok=True) - temp_path: Path | None = None - try: - with tempfile.NamedTemporaryFile( - "w", - encoding="utf-8", - dir=path.parent, - prefix=f".{path.name}.", - delete=False, - ) as file: - temp_path = Path(file.name) - for question_id in order: - if question_id not in results: - continue - row = {"question_id": question_id, **results[question_id]} - file.write( - json.dumps(row, ensure_ascii=False, separators=(",", ":")) + "\n", - ) - file.flush() - os.fsync(file.fileno()) - os.replace(temp_path, path) - finally: - if temp_path is not None and temp_path.exists(): - temp_path.unlink() - - -def run_one( - question_id: str, - workspace: Path, - log_dir: Path, - reference_paths: list[Path], -) -> dict[str, Any]: - """Run the configured one-shot job and validate its stdout JSON.""" - env = dict(os.environ, LME_WORKSPACE_DIR=str(workspace.relative_to(REPO))) - env[REFERENCE_PATHS_ENV] = json.dumps( - [str(path.resolve()) for path in reference_paths], - ensure_ascii=False, - ) - completed = subprocess.run( - [ - sys.executable, - "-c", - "from reme.reme import main; main()", - "start", - "config=jinli_lme", - "job=final_answer_review", - ], - cwd=REPO, - env=env, - text=True, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - check=False, - ) - log_dir.mkdir(parents=True, exist_ok=True) - log_path = log_dir / f"{question_id}.log" - log_text = ( - f"workspace={workspace}\nreturncode={completed.returncode}\n\n" - f"[stdout]\n{completed.stdout}\n[stderr]\n{completed.stderr}" - ) - log_path.write_text( - log_text, - encoding="utf-8", - ) - if completed.returncode != 0: - raise RuntimeError( - f"Job failed for {question_id} with rc={completed.returncode}; see {log_path}", - ) - try: - value = json.loads(completed.stdout.strip()) - except json.JSONDecodeError as exc: - raise ValueError( - f"Job stdout is not JSON for {question_id}; see {log_path}", - ) from exc - return _validate_result(value, source=f"job result for {question_id}") - - -def main() -> int: - """Review and checkpoint the selected dataset cases sequentially.""" - args = parse_args() - if args.limit < 0: - raise ValueError("--limit must be >= 0") - if not 1 <= args.concurrency <= MAX_CONCURRENCY: - raise ValueError(f"--concurrency must be between 1 and {MAX_CONCURRENCY}") - if args.submit_interval_seconds <= MIN_SUBMIT_INTERVAL_SECONDS: - raise ValueError( - f"--submit-interval-seconds must be > {MIN_SUBMIT_INTERVAL_SECONDS:g}", - ) - - reference_paths = [path.resolve() for path in (args.references or DEFAULT_REFERENCES)] - mapping = workspace_map() - references = merge_references(reference_paths) - missing = [question_id for question_id in references if question_id not in mapping] - if missing: - raise ValueError(f"No dataset workspace for question IDs: {', '.join(missing)}") - - full_order = list(mapping) - excluded = set(references) if args.exclude_reference_question_ids else set() - order = select_question_ids(mapping, args.question_ids, excluded) - if args.only_reference_question_ids: - order = [question_id for question_id in order if question_id in references] - results = {} if args.no_resume else load_existing(args.output.resolve()) - pending = ( - list(order) if args.rerun_selected else [question_id for question_id in order if question_id not in results] - ) - if args.limit: - pending = pending[: args.limit] - - no_reference = sum(question_id not in references for question_id in order) - one_reference = sum(len(references.get(question_id, [])) == 1 for question_id in order) - multiple_references = sum(len(references.get(question_id, [])) > 1 for question_id in order) - print( - f"total={len(order)} no_reference={no_reference} one_reference={one_reference} " - f"multiple_references={multiple_references} " - f"excluded={len(excluded)} " - f"only_reference_questions={args.only_reference_question_ids} " - f"concurrency={args.concurrency} submit_interval={args.submit_interval_seconds:g}s " - f"existing={len(results)} pending={len(pending)} output={args.output.resolve()}", - flush=True, - ) - - if args.dry_run: - for question_id in pending: - print( - f"[would-run] question_id={question_id} workspace={mapping[question_id].name} " - f"references={len(references.get(question_id, []))}", - ) - return 0 - - executor = concurrent.futures.ThreadPoolExecutor(max_workers=args.concurrency) - active: dict[concurrent.futures.Future[dict[str, Any]], tuple[int, str]] = {} - next_position = 0 - saved_count = 0 - next_submit_at = 0.0 - try: - while next_position < len(pending) or active: - can_submit = next_position < len(pending) and len(active) < args.concurrency - if can_submit and time.monotonic() >= next_submit_at: - question_id = pending[next_position] - position = next_position + 1 - workspace = mapping[question_id] - print( - f"[submit {position}/{len(pending)}] question_id={question_id} " - f"workspace={workspace.name} references={len(references.get(question_id, []))}", - flush=True, - ) - future = executor.submit( - run_one, - question_id, - workspace, - args.log_dir.resolve(), - reference_paths, - ) - active[future] = (position, question_id) - next_position += 1 - next_submit_at = time.monotonic() + args.submit_interval_seconds - continue - - if not active: - time.sleep(max(0.0, next_submit_at - time.monotonic())) - continue - - timeout = None - if can_submit: - timeout = max(0.0, next_submit_at - time.monotonic()) - done, _ = concurrent.futures.wait( - active, - timeout=timeout, - return_when=concurrent.futures.FIRST_COMPLETED, - ) - for future in done: - position, question_id = active.pop(future) - results[question_id] = future.result() - atomic_write_results(args.output.resolve(), full_order, results) - saved_count += 1 - print( - f"[saved {saved_count}/{len(pending)}] submitted_position={position} " f"question_id={question_id}", - flush=True, - ) - finally: - executor.shutdown(wait=True, cancel_futures=True) - - print( - f"ALL FINISHED total_saved={sum(question_id in results for question_id in order)}", - flush=True, - ) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/benchmark/longmemeval/run_golden_check.py b/benchmark/longmemeval/run_golden_check.py deleted file mode 100644 index 18e4d847..00000000 --- a/benchmark/longmemeval/run_golden_check.py +++ /dev/null @@ -1,216 +0,0 @@ -#!/usr/bin/env python3 -"""Run LongMemEval ``golden_check`` concurrently across samples. - -For every workspace under ``datasets/longmemeval/`` in the selected numeric -range, this launches: - - reme start config=jinli_lme job=golden_check - -with ``LME_WORKSPACE_DIR`` pointed at that sample. Multiple samples can run at -once, capped by ``--concurrency``. The ``golden_check`` job itself waits for -``session_review.json`` when configured with ``wait_for_paths_step`` in -``jinli_lme.yaml``. Each sample's stdout/stderr goes to -``logs/golden_check/.log``. - -By default the script processes samples 0..499 inclusive and reruns every sample -in that range. Pass ``--resume`` to skip samples whose ``check_golden.json`` -already exists. - -Examples: - python benchmark/longmemeval/run_golden_check.py - python benchmark/longmemeval/run_golden_check.py --start 187 --end 499 - python benchmark/longmemeval/run_golden_check.py --concurrency 8 --stagger 1 - python benchmark/longmemeval/run_golden_check.py --progress-interval 10 - python benchmark/longmemeval/run_golden_check.py --resume - python benchmark/longmemeval/run_golden_check.py --limit 5 --dry-run -""" - -import argparse -import asyncio -import json -import os -import time -from pathlib import Path - -REPO = Path(__file__).resolve().parents[2] -DATA = REPO / "datasets" / "longmemeval" -LOGDIR = REPO / "logs" / "golden_check" -OUTPUT_FILENAME = "check_golden.json" - - -def parse_args() -> argparse.Namespace: - """Parse command-line arguments.""" - p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - p.add_argument("--start", type=int, default=0, help="first numeric sample id to process, inclusive (default 0)") - p.add_argument("--end", type=int, default=499, help="last numeric sample id to process, inclusive (default 499)") - p.add_argument("--limit", type=int, default=0, help="only process the first N selected samples (0 = all)") - p.add_argument("--concurrency", type=int, default=3, help="max samples running at once (default 3)") - p.add_argument("--stagger", type=float, default=1.0, help="seconds between consecutive launches (default 1)") - p.add_argument( - "--progress-interval", - type=float, - default=30.0, - help="seconds between progress reports while running (0 = disabled, default 30)", - ) - p.add_argument( - "--resume", - action="store_true", - help=f"skip samples whose {OUTPUT_FILENAME} already exists", - ) - p.add_argument("--dry-run", action="store_true", help="list what would run, launch nothing") - return p.parse_args() - - -def sample_ids() -> list[str]: - """List all sample IDs (numeric workspace dirs), numerically sorted.""" - ids = [p.name for p in DATA.iterdir() if p.is_dir() and p.name.isdigit()] - return sorted(ids, key=int) - - -def output_is_current(idx: str) -> bool: - """Return True when the sample already has a current-schema golden-check artifact.""" - path = DATA / idx / OUTPUT_FILENAME - try: - with path.open(encoding="utf-8") as f: - data = json.load(f) - except (OSError, json.JSONDecodeError): - return False - verdict = data.get("verdict") if isinstance(data, dict) else None - if not isinstance(verdict, dict): - return False - return isinstance(verdict.get("golden_answer_correct"), bool) and isinstance( - verdict.get("answer_session_ids_correct"), - bool, - ) - - -def print_progress(counters: dict, active: set[str], selected_total: int, started_at: float) -> None: - """Print a one-line progress snapshot.""" - finished = counters["done"] + counters["fail"] + counters["skip"] - running = len(active) - outstanding = max(selected_total - finished - running, 0) - elapsed = time.monotonic() - started_at - print( - f"[progress] selected={selected_total} done={counters['done']} fail={counters['fail']} " - f"skip={counters['skip']} running={running} outstanding={outstanding} " - f"elapsed={elapsed:.0f}s", - flush=True, - ) - - -async def progress_reporter( - counters: dict, - active: set[str], - selected_total: int, - started_at: float, - interval: float, - stop: asyncio.Event, -) -> None: - """Periodically report progress until ``stop`` is set.""" - if interval <= 0: - return - while not stop.is_set(): - try: - await asyncio.wait_for(stop.wait(), timeout=interval) - except asyncio.TimeoutError: - print_progress(counters, active, selected_total, started_at) - - -async def run_one(idx: str, sem: asyncio.Semaphore, resume: bool, counters: dict, active: set[str]) -> None: - """Run ``golden_check`` for one sample.""" - if resume and output_is_current(idx): - counters["skip"] += 1 - print(f"[skip] {idx} ({OUTPUT_FILENAME} exists)", flush=True) - return - - async with sem: - active.add(idx) - log = LOGDIR / f"{idx}.log" - log.parent.mkdir(parents=True, exist_ok=True) - env = dict(os.environ, LME_WORKSPACE_DIR=f"datasets/longmemeval/{idx}") - - started = time.strftime("%H:%M:%S") - print(f"[start {started}] {idx}", flush=True) - try: - with log.open("w", encoding="utf-8") as f: - proc = await asyncio.create_subprocess_exec( - "reme", - "start", - "config=jinli_lme", - "job=golden_check", - cwd=str(REPO), - env=env, - stdout=f, - stderr=asyncio.subprocess.STDOUT, - ) - rc = await proc.wait() - - ok = rc == 0 and output_is_current(idx) - counters["done" if ok else "fail"] += 1 - tag = "done" if ok else "fail" - print( - f"[{tag}] {idx} rc={rc} log={log} ({counters['done']} done / {counters['fail']} fail)", - flush=True, - ) - finally: - active.discard(idx) - - -async def main() -> int: - """Run the concurrent driver.""" - args = parse_args() - if args.end < args.start: - raise ValueError(f"--end ({args.end}) must be >= --start ({args.start})") - if args.concurrency < 1: - raise ValueError("--concurrency must be >= 1") - if args.progress_interval < 0: - raise ValueError("--progress-interval must be >= 0") - - LOGDIR.mkdir(parents=True, exist_ok=True) - - ids = [i for i in sample_ids() if args.start <= int(i) <= args.end] - if args.limit: - ids = ids[: args.limit] - - pending = [i for i in ids if not (args.resume and output_is_current(i))] - print( - f"job=golden_check samples total={len(ids)} pending={len(pending)} " - f"range={args.start}..{args.end} resume={args.resume} " - f"concurrency={args.concurrency} stagger={args.stagger}s", - flush=True, - ) - - if args.dry_run: - for idx in pending: - print(f"[would-run] {idx}") - return 0 - - sem = asyncio.Semaphore(args.concurrency) - counters = {"done": 0, "fail": 0, "skip": 0} - active: set[str] = set() - started_at = time.monotonic() - stop_progress = asyncio.Event() - progress_task = asyncio.create_task( - progress_reporter(counters, active, len(ids), started_at, args.progress_interval, stop_progress), - ) - tasks: list[asyncio.Task] = [] - try: - for n, idx in enumerate(ids): - if n and args.stagger > 0: - await asyncio.sleep(args.stagger) - tasks.append(asyncio.create_task(run_one(idx, sem, args.resume, counters, active))) - - await asyncio.gather(*tasks) - finally: - stop_progress.set() - await progress_task - print_progress(counters, active, len(ids), started_at) - print( - f"ALL FINISHED done={counters['done']} fail={counters['fail']} skip={counters['skip']}", - flush=True, - ) - return 0 if counters["fail"] == 0 else 1 - - -if __name__ == "__main__": - raise SystemExit(asyncio.run(main())) diff --git a/benchmark/longmemeval/run_session_review.py b/benchmark/longmemeval/run_session_review.py deleted file mode 100644 index c40b03b8..00000000 --- a/benchmark/longmemeval/run_session_review.py +++ /dev/null @@ -1,203 +0,0 @@ -#!/usr/bin/env python3 -"""Run LongMemEval ``session_review`` concurrently across samples. - -For every workspace under ``datasets/longmemeval/`` in the selected numeric -range, this launches: - - reme start config=jinli_lme job=session_review - -with ``LME_WORKSPACE_DIR`` pointed at that sample. Multiple samples can run at -once, capped by ``--concurrency``. By default this runner launches one sample at -a time; request submission is throttled inside each ``session_review`` process. -Each sample's stdout/stderr goes to ``logs/session_review/.log``. - -By default the script processes samples 0..499 inclusive and reruns every sample -in that range. Pass ``--resume`` to skip samples whose ``session_review.json`` -already exists. - -Examples: - python benchmark/longmemeval/run_session_review.py - python benchmark/longmemeval/run_session_review.py --start 187 --end 499 - python benchmark/longmemeval/run_session_review.py --concurrency 2 - python benchmark/longmemeval/run_session_review.py --resume - python benchmark/longmemeval/run_session_review.py --limit 5 --dry-run -""" - -import argparse -import asyncio -import json -import os -import time -from pathlib import Path - -REPO = Path(__file__).resolve().parents[2] -DATA = REPO / "datasets" / "longmemeval" -LOGDIR = REPO / "logs" / "session_review" -OUTPUT_FILENAME = "session_review.json" - - -def parse_args() -> argparse.Namespace: - """Parse command-line arguments.""" - p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - p.add_argument("--start", type=int, default=0, help="first numeric sample id to process, inclusive (default 0)") - p.add_argument("--end", type=int, default=499, help="last numeric sample id to process, inclusive (default 499)") - p.add_argument("--limit", type=int, default=0, help="only process the first N selected samples (0 = all)") - p.add_argument("--concurrency", type=int, default=1, help="max samples running at once (default 1)") - p.add_argument("--stagger", type=float, default=1.0, help="seconds between worker launches (default 1)") - p.add_argument( - "--resume", - action="store_true", - help=f"skip samples whose {OUTPUT_FILENAME} already exists", - ) - p.add_argument("--dry-run", action="store_true", help="list what would run, launch nothing") - p.add_argument("--stop-on-fail", action="store_true", help="stop immediately after the first failed sample") - return p.parse_args() - - -def sample_ids() -> list[str]: - """List all sample IDs (numeric workspace dirs), numerically sorted.""" - ids = [p.name for p in DATA.iterdir() if p.is_dir() and p.name.isdigit()] - return sorted(ids, key=int) - - -def output_exists(idx: str) -> bool: - """Return True when the sample already has a session review artifact.""" - return (DATA / idx / OUTPUT_FILENAME).exists() - - -def output_is_healthy(idx: str) -> bool: - """Return True when ``session_review.json`` exists and has no failed reviews.""" - path = DATA / idx / OUTPUT_FILENAME - if not path.exists(): - return False - try: - with path.open(encoding="utf-8") as f: - data = json.load(f) - except (OSError, json.JSONDecodeError): - return False - review = data.get("review") if isinstance(data, dict) else None - if not isinstance(review, dict): - return False - raw = review.get("num_failed_reviews") - if isinstance(raw, int): - return raw == 0 - failed_reviews = review.get("failed_reviews") - return not failed_reviews - - -async def run_one(idx: str, active: set[str]) -> bool: - """Run ``session_review`` for one sample. Returns True on success.""" - log = LOGDIR / f"{idx}.log" - log.parent.mkdir(parents=True, exist_ok=True) - env = dict(os.environ, LME_WORKSPACE_DIR=f"datasets/longmemeval/{idx}") - - started = time.strftime("%H:%M:%S") - print(f"[start {started}] {idx}", flush=True) - active.add(idx) - try: - with log.open("w", encoding="utf-8") as f: - proc = await asyncio.create_subprocess_exec( - "reme", - "start", - "config=jinli_lme", - "job=session_review", - cwd=str(REPO), - env=env, - stdout=f, - stderr=asyncio.subprocess.STDOUT, - ) - rc = await proc.wait() - finally: - active.discard(idx) - - ok = rc == 0 and output_exists(idx) - tag = "done" if ok else "fail" - print(f"[{tag}] {idx} rc={rc} log={log}", flush=True) - return ok - - -async def worker( - name: int, - queue: asyncio.Queue[str], - args: argparse.Namespace, - counters: dict[str, int], - active: set[str], - stop: asyncio.Event, -) -> None: - """Run samples from ``queue`` until exhausted or fail-fast is triggered.""" - if name and args.stagger > 0: - await asyncio.sleep(args.stagger * name) - - while not stop.is_set(): - try: - idx = queue.get_nowait() - except asyncio.QueueEmpty: - return - - try: - if args.resume and output_is_healthy(idx): - counters["skip"] += 1 - print(f"[skip] {idx} (healthy {OUTPUT_FILENAME} exists)", flush=True) - continue - - if await run_one(idx, active): - counters["done"] += 1 - else: - counters["fail"] += 1 - if args.stop_on_fail: - stop.set() - finally: - queue.task_done() - - -async def main() -> int: - """Run the concurrent driver.""" - args = parse_args() - if args.end < args.start: - raise ValueError(f"--end ({args.end}) must be >= --start ({args.start})") - if args.concurrency < 1: - raise ValueError("--concurrency must be >= 1") - if args.stagger < 0: - raise ValueError("--stagger must be >= 0") - - LOGDIR.mkdir(parents=True, exist_ok=True) - - ids = [i for i in sample_ids() if args.start <= int(i) <= args.end] - if args.limit: - ids = ids[: args.limit] - - pending = [i for i in ids if not (args.resume and output_exists(i))] - print( - f"job=session_review samples total={len(ids)} pending={len(pending)} " - f"range={args.start}..{args.end} resume={args.resume} " - f"concurrency={args.concurrency} stagger={args.stagger}s", - flush=True, - ) - - if args.dry_run: - for idx in pending: - print(f"[would-run] {idx}") - return 0 - - counters: dict[str, int] = {"done": 0, "fail": 0, "skip": 0} - active: set[str] = set() - stop = asyncio.Event() - queue: asyncio.Queue[str] = asyncio.Queue() - for idx in ids: - queue.put_nowait(idx) - - workers = [ - asyncio.create_task(worker(n, queue, args, counters, active, stop)) - for n in range(min(args.concurrency, len(ids))) - ] - await asyncio.gather(*workers) - - print( - f"ALL FINISHED done={counters['done']} fail={counters['fail']} skip={counters['skip']}", - flush=True, - ) - return 0 if counters["fail"] == 0 else 1 - - -if __name__ == "__main__": - raise SystemExit(asyncio.run(main())) diff --git a/benchmark/longmemeval/stats_agentic_answer.py b/benchmark/longmemeval/stats_agentic_answer.py deleted file mode 100644 index 37ddea6a..00000000 --- a/benchmark/longmemeval/stats_agentic_answer.py +++ /dev/null @@ -1,202 +0,0 @@ -#!/usr/bin/env python3 -"""Summarise the ``agentic_answer`` results across all LongMemEval samples. - -Reports progress (how many of the 500 samples produced ``mem_answer.json``) and a -breakdown of answer *status*: - - answered — a non-empty answer that is not "not provided"; - - not_provided — the agent gave up ("not provided"); - - empty — ``mem_answer.json`` exists but the answer is blank; - - missing — no ``mem_answer.json`` yet. - -Everything is broken down by ``question_type``. This script does NOT judge answer -correctness (there is no grader for ``mem_answer`` yet) — it only tracks progress -and collects predicted-vs-golden pairs. Tool-call statistics are read from the -aggregate written by ``run_agentic_answer.py`` when it is present. - -Examples: - python benchmark/longmemeval/stats_agentic_answer.py - python benchmark/longmemeval/stats_agentic_answer.py --list-run-failed - python benchmark/longmemeval/stats_agentic_answer.py --list-unanswered - python benchmark/longmemeval/stats_agentic_answer.py --json -""" - -import argparse -import json -from collections import defaultdict -from pathlib import Path - -REPO = Path(__file__).resolve().parents[2] -DATA = REPO / "datasets" / "longmemeval" -LOGBASE = REPO / "logs" / "agentic_answer" -AGGREGATE = LOGBASE / "aggregate.json" - - -def parse_args() -> argparse.Namespace: - """Parse command-line arguments.""" - p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - p.add_argument("--list-unanswered", action="store_true", help="list samples answered 'not provided' or empty") - p.add_argument("--list-run-failed", action="store_true", help="list launched samples with no readable output") - p.add_argument("--json", action="store_true", help="emit the summary as JSON") - return p.parse_args() - - -def sample_ids() -> list[str]: - """List all sample IDs (numeric workspace dirs), numerically sorted.""" - ids = [p.name for p in DATA.iterdir() if p.is_dir() and p.name.isdigit()] - return sorted(ids, key=int) - - -def pct(num: int, den: int) -> str: - """Format a percentage.""" - return f"{(100.0 * num / den):.1f}%" if den else "n/a" - - -def logged_sample_ids() -> list[str]: - """List sample IDs that have an agentic_answer launch log.""" - logdir = LOGBASE / "agentic_answer" - if not logdir.exists(): - return [] - ids = [p.stem for p in logdir.glob("*.log") if p.stem.isdigit()] - return sorted(ids, key=int) - - -def answer_status(pred: str, has_file: bool) -> str: - """Classify an answer into answered / not_provided / empty / missing.""" - if not has_file: - return "missing" - if not pred: - return "empty" - if "not provided" in pred.lower(): - return "not_provided" - return "answered" - - -def load_tool_calls() -> dict[str, int]: - """Map idx -> num_tool_calls from the aggregate, if it exists.""" - if not AGGREGATE.exists(): - return {} - try: - with AGGREGATE.open(encoding="utf-8") as f: - data = json.load(f) - except (OSError, json.JSONDecodeError): - return {} - return {s["idx"]: s.get("num_tool_calls", 0) for s in data.get("samples", []) if "idx" in s} - - -def main() -> int: - """Main entry point.""" - args = parse_args() - ids = sample_ids() - total = len(ids) - tool_calls = load_tool_calls() - - rows, unreadable = [], [] - finished_ids = set() - for idx in ids: - query_path = DATA / idx / "query.json" - mem_path = DATA / idx / "mem_answer.json" - qtype = "(unknown)" - try: - with query_path.open(encoding="utf-8") as f: - qtype = json.load(f).get("question_type") or "(unknown)" - except (OSError, json.JSONDecodeError): - pass - - has_file = mem_path.exists() - pred = "" - if has_file: - try: - with mem_path.open(encoding="utf-8") as f: - pred = str(json.load(f).get("answer") or "").strip() - finished_ids.add(idx) - except (OSError, json.JSONDecodeError): - unreadable.append(idx) - has_file = False - - rows.append({"idx": idx, "type": qtype, "status": answer_status(pred, has_file)}) - - finished = [r for r in rows if r["status"] != "missing"] - n = len(finished) - launched = logged_sample_ids() - run_failed = [idx for idx in launched if idx not in finished_ids] - - # Overall status tallies. - status_counts: dict[str, int] = defaultdict(int) - for r in rows: - status_counts[r["status"]] += 1 - answered = status_counts["answered"] - unanswered = [r["idx"] for r in rows if r["status"] in ("not_provided", "empty")] - - calls_vals = [tool_calls[i] for i in finished_ids if i in tool_calls] - avg_calls = sum(calls_vals) / len(calls_vals) if calls_vals else 0.0 - - # Per question_type breakdown. - by_type: dict[str, dict[str, int]] = defaultdict(lambda: {"n": 0, "answered": 0}) - for r in finished: - by_type[r["type"]]["n"] += 1 - by_type[r["type"]]["answered"] += 1 if r["status"] == "answered" else 0 - - if args.json: - print( - json.dumps( - { - "total": total, - "finished": n, - "pending": total - n - len(unreadable), - "unreadable": unreadable, - "launched": len(launched), - "run_failed": run_failed, - "status_counts": dict(status_counts), - "answered_rate": round(answered / n, 4) if n else None, - "avg_tool_calls": round(avg_calls, 2) if calls_vals else None, - "by_type": { - t: {**c, "answered_rate": round(c["answered"] / c["n"], 4)} for t, c in by_type.items() - }, - "unanswered": unanswered, - "aggregate": str(AGGREGATE) if AGGREGATE.exists() else None, - }, - ensure_ascii=False, - indent=2, - ), - ) - return 0 - - print("=" * 60) - print("LongMemEval agentic_answer 统计") - print("=" * 60) - print(f"样例总数 : {total}") - print(f"已完成 (有产出) : {n} ({pct(n, total)})") - print(f"未完成 : {total - n - len(unreadable)}") - if unreadable: - print(f"损坏/无法解析 : {len(unreadable)} {unreadable}") - print(f"已启动过 (有 log) : {len(launched)}") - print(f"运行失败/无可读产出 : {len(run_failed)}") - print("-" * 60) - print(f"已作答 (非 not provided): {answered} ({pct(answered, n)} of finished)") - print(f" 其中 not provided : {status_counts['not_provided']}") - print(f" 其中 空答案 : {status_counts['empty']}") - if calls_vals: - print(f"平均工具调用次数 : {avg_calls:.1f} (来自 {AGGREGATE.name})") - else: - print("平均工具调用次数 : n/a (先跑 run_agentic_answer.py 生成 aggregate.json)") - print("-" * 60) - print("按 question_type:") - print(f" {'type':<24} {'n':>4} {'已作答率':>12}") - for t in sorted(by_type): - c = by_type[t] - print(f" {t:<24} {c['n']:>4} {pct(c['answered'], c['n']):>12}") - - if args.list_unanswered: - print("-" * 60) - print(f"not provided / 空答案的样例 ({len(unanswered)}): {unanswered}") - if args.list_run_failed: - print("-" * 60) - print(f"运行失败/无可读 mem_answer.json 的样例 ({len(run_failed)}): {run_failed}") - for idx in run_failed: - print(f" {idx}: {LOGBASE / 'agentic_answer' / f'{idx}.log'}") - print("=" * 60) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/benchmark/longmemeval/stats_golden_check.py b/benchmark/longmemeval/stats_golden_check.py deleted file mode 100644 index 97f7f7f6..00000000 --- a/benchmark/longmemeval/stats_golden_check.py +++ /dev/null @@ -1,344 +0,0 @@ -#!/usr/bin/env python3 -"""Summarise the ``check_golden.json`` verdicts across all LongMemEval samples. - -Reports progress (how many of the 500 samples have finished) and accuracy: - - golden answer accuracy = share of finished samples whose golden answer the - auditor judged correct (``verdict.golden_answer_correct``); - - answer_session_ids accuracy = share whose claimed answer sessions the auditor - judged exactly correct (``verdict.answer_session_ids_correct``). - -Everything is also broken down by ``question_type``. Use ``--list-bad`` to print -the samples whose golden answer was judged NOT correct. - -Examples: - python benchmark/longmemeval/stats_golden_check.py - python benchmark/longmemeval/stats_golden_check.py --list-bad - python benchmark/longmemeval/stats_golden_check.py --list-run-failed - python benchmark/longmemeval/stats_golden_check.py --json -""" - -import argparse -import json -from collections import defaultdict -from pathlib import Path - -REPO = Path(__file__).resolve().parents[2] -DATA = REPO / "datasets" / "longmemeval" -LOGDIR = REPO / "logs" / "golden_check" - - -def parse_args() -> argparse.Namespace: - """Parse command-line arguments.""" - p = argparse.ArgumentParser(description=__doc__) - p.add_argument("--list-bad", action="store_true", help="list samples whose golden answer is NOT correct") - p.add_argument( - "--list-bad-sessions", - action="store_true", - help="list samples whose answer_session_ids is NOT correct", - ) - p.add_argument( - "--list-run-failed", - action="store_true", - help="list launched samples that did not produce readable output", - ) - p.add_argument("--json", action="store_true", help="emit the summary as JSON") - return p.parse_args() - - -def sample_ids() -> list[str]: - """List all sample IDs.""" - ids = [p.name for p in DATA.iterdir() if p.is_dir() and p.name.isdigit()] - return sorted(ids, key=int) - - -def pct(num: int, den: int) -> str: - """Format a percentage.""" - return f"{(100.0 * num / den):.1f}%" if den else "n/a" - - -def logged_sample_ids() -> list[str]: - """List all sample IDs that have been launched but not finished.""" - if not LOGDIR.exists(): - return [] - ids = [p.stem for p in LOGDIR.glob("*.log") if p.stem.isdigit()] - return sorted(ids, key=int) - - -def load_json(path: Path) -> dict: - """Load a JSON object, returning {} on any error.""" - try: - with path.open(encoding="utf-8") as f: - data = json.load(f) - return data if isinstance(data, dict) else {} - except (OSError, json.JSONDecodeError): - return {} - - -def question_type_for(idx: str, data: dict) -> str: - """Return question_type from the output, session review, or query.json.""" - question_type = str(data.get("question_type") or "").strip() - if question_type: - return question_type - - review_path_raw = str(data.get("session_review_path") or "").strip() - review_path = Path(review_path_raw) if review_path_raw else DATA / idx / "session_review.json" - if not review_path.is_absolute(): - review_path = REPO / review_path - review = load_json(review_path) - review_question_type = str((review.get("query") or {}).get("question_type") or "").strip() - if review_question_type: - return review_question_type - - query = load_json(DATA / idx / "query.json") - return str(query.get("question_type") or "(unknown)").strip() or "(unknown)" - - -def question_id_for(idx: str, data: dict) -> str: - """Return question_id from the output, session review, or query.json.""" - question_id = str(data.get("question_id") or "").strip() - if question_id: - return question_id - - review_path_raw = str(data.get("session_review_path") or "").strip() - review_path = Path(review_path_raw) if review_path_raw else DATA / idx / "session_review.json" - if not review_path.is_absolute(): - review_path = REPO / review_path - review = load_json(review_path) - review_question_id = str((review.get("query") or {}).get("question_id") or "").strip() - if review_question_id: - return review_question_id - - query = load_json(DATA / idx / "query.json") - return str(query.get("question_id") or "").strip() - - -def sample_label(data: dict) -> str: - """Format sample id as idx(question_id) when question_id is available.""" - idx = str(data.get("_idx") or "") - qid = str(data.get("_question_id") or "").strip() - return f"{idx}({qid})" if qid else idx - - -def related_session_ids(data: dict) -> list[str]: - """Return the best available session ids for a bad verdict record.""" - verdict = data.get("verdict") if isinstance(data, dict) else None - if isinstance(verdict, dict): - true_ids = verdict.get("true_answer_session_ids") - if isinstance(true_ids, list): - ids = [str(session_id) for session_id in true_ids if str(session_id).strip()] - if ids: - return ids - - summaries = data.get("session_summaries") - if isinstance(summaries, list): - return [ - str(summary.get("session_id")) - for summary in summaries - if isinstance(summary, dict) and str(summary.get("session_id") or "").strip() - ] - return [] - - -def grouped_records(records: list[dict]) -> dict[str, list[dict]]: - """Group records by question_type for human-readable list output.""" - grouped: dict[str, list[dict]] = defaultdict(list) - for data in records: - question_type = str(data.get("_question_type") or "(unknown)") - grouped[question_type].append( - { - "index": str(data.get("_idx") or ""), - "question_id": str(data.get("_question_id") or ""), - "session_id": related_session_ids(data), - }, - ) - return dict(sorted(grouped.items())) - - -def verdict_bool(verdict: dict, new_key: str, old_key: str) -> bool: - """Read a verdict boolean, accepting the old field name for compatibility.""" - if verdict.get(new_key) is True: - return True - if verdict.get(new_key) is False: - return False - return verdict.get(old_key) is True - - -def has_current_verdict(data: dict) -> bool: - """Return True when ``check_golden.json`` uses the current golden_check schema.""" - verdict = data.get("verdict") if isinstance(data, dict) else None - if not isinstance(verdict, dict): - return False - return isinstance(verdict.get("golden_answer_correct"), bool) and isinstance( - verdict.get("answer_session_ids_correct"), - bool, - ) - - -def write_golden_check_list(done: list[dict], output_path: Path) -> None: - """Write all readable check_golden records as JSONL.""" - with output_path.open("w", encoding="utf-8") as f: - for data in done: - f.write(json.dumps(data, ensure_ascii=False)) - f.write("\n") - - -def main() -> int: - """Main entry point.""" - args = parse_args() - ids = sample_ids() - total = len(ids) - - done, unreadable, stale = [], [], [] - finished_ids = set() - for idx in ids: - path = DATA / idx / "check_golden.json" - if not path.exists(): - continue - try: - with path.open(encoding="utf-8") as f: - data = json.load(f) - if not has_current_verdict(data): - stale.append(idx) - continue - data["_idx"] = idx - data["_question_type"] = question_type_for(idx, data) - data["_question_id"] = question_id_for(idx, data) - done.append(data) - finished_ids.add(idx) - except (OSError, json.JSONDecodeError): - unreadable.append(idx) - - n = len(done) - output_path = Path.cwd() / "golden_check_list.jsonl" - write_golden_check_list(done, output_path) - launched = logged_sample_ids() - run_failed = [idx for idx in launched if idx not in finished_ids] - - # Overall tallies. - golden_ok = sum( - 1 for d in done if verdict_bool(d.get("verdict", {}), "golden_answer_correct", "golden_answer_reasonable") - ) - sess_ok = sum( - 1 - for d in done - if verdict_bool(d.get("verdict", {}), "answer_session_ids_correct", "answer_session_ids_reasonable") - ) - both_ok = sum( - 1 - for d in done - if verdict_bool(d.get("verdict", {}), "golden_answer_correct", "golden_answer_reasonable") - and verdict_bool(d.get("verdict", {}), "answer_session_ids_correct", "answer_session_ids_reasonable") - ) - - # Per question_type breakdown. - by_type: dict[str, dict[str, int]] = defaultdict(lambda: {"n": 0, "golden_ok": 0, "sess_ok": 0, "both_ok": 0}) - for d in done: - v = d.get("verdict", {}) - golden_is_ok = verdict_bool(v, "golden_answer_correct", "golden_answer_reasonable") - sess_is_ok = verdict_bool(v, "answer_session_ids_correct", "answer_session_ids_reasonable") - t = d.get("_question_type") or "(unknown)" - by_type[t]["n"] += 1 - by_type[t]["golden_ok"] += 1 if golden_is_ok else 0 - by_type[t]["sess_ok"] += 1 if sess_is_ok else 0 - by_type[t]["both_ok"] += 1 if golden_is_ok and sess_is_ok else 0 - - bad_golden_records = [ - d for d in done if not verdict_bool(d.get("verdict", {}), "golden_answer_correct", "golden_answer_reasonable") - ] - bad_session_records = [ - d - for d in done - if not verdict_bool(d.get("verdict", {}), "answer_session_ids_correct", "answer_session_ids_reasonable") - ] - bad_golden = [d["_idx"] for d in bad_golden_records] - bad_sessions = [d["_idx"] for d in bad_session_records] - - if args.json: - print( - json.dumps( - { - "total": total, - "finished": n, - "pending": total - n - len(unreadable), - "unreadable": unreadable, - "stale": stale, - "launched": len(launched), - "run_failed": run_failed, - "golden_answer_accuracy": round(golden_ok / n, 4) if n else None, - "answer_session_ids_accuracy": round(sess_ok / n, 4) if n else None, - "both_correct_rate": round(both_ok / n, 4) if n else None, - "golden_ok": golden_ok, - "sess_ok": sess_ok, - "both_ok": both_ok, - "by_type": { - t: { - **c, - "golden_bad": c["n"] - c["golden_ok"], - "session_bad": c["n"] - c["sess_ok"], - "both_bad": c["n"] - c["both_ok"], - "golden_acc": round(c["golden_ok"] / c["n"], 4), - "session_acc": round(c["sess_ok"] / c["n"], 4), - "both_acc": round(c["both_ok"] / c["n"], 4), - } - for t, c in by_type.items() - }, - "bad_golden": bad_golden, - "bad_sessions": bad_sessions, - "golden_check_list": str(output_path), - }, - ensure_ascii=False, - indent=2, - ), - ) - return 0 - - print("=" * 60) - print("LongMemEval golden_check 统计") - print("=" * 60) - print(f"样例总数 : {total}") - print(f"已完成 (有产出) : {n} ({pct(n, total)})") - print(f"未完成 : {total - n - len(unreadable)}") - if unreadable: - print(f"损坏/无法解析 : {len(unreadable)} {unreadable}") - if stale: - print(f"旧格式待重跑 : {len(stale)} {stale}") - print(f"已合并 JSONL : {output_path}") - print(f"已启动过 (有 log) : {len(launched)}") - print(f"运行失败/无可读产出 : {len(run_failed)}") - print("-" * 60) - print(f"golden answer 正确率 : {pct(golden_ok, n)} ({golden_ok}/{n})") - print(f"answer_session 正确率: {pct(sess_ok, n)} ({sess_ok}/{n})") - print(f"两者都正确 : {pct(both_ok, n)} ({both_ok}/{n})") - print("-" * 60) - print("按 question_type:") - print( - f" {'type':<24} {'n':>4} {'golden正确率':>14} {'golden错误':>10} " - f"{'session正确率':>14} {'session错误':>11} {'都正确':>10} {'都正确错误':>12}", - ) - for t in sorted(by_type): - c = by_type[t] - print( - f" {t:<24} {c['n']:>4} {pct(c['golden_ok'], c['n']):>14} {c['n'] - c['golden_ok']:>10} " - f"{pct(c['sess_ok'], c['n']):>14} {c['n'] - c['sess_ok']:>11} " - f"{pct(c['both_ok'], c['n']):>10} {c['n'] - c['both_ok']:>12}", - ) - - if args.list_bad: - print("-" * 60) - print(f"golden answer 判为不正确的样例 ({len(bad_golden_records)}):") - print(json.dumps(grouped_records(bad_golden_records), ensure_ascii=False)) - if args.list_bad_sessions: - print("-" * 60) - print(f"answer_session_ids 判为不正确的样例 ({len(bad_session_records)}):") - print(json.dumps(grouped_records(bad_session_records), ensure_ascii=False)) - if args.list_run_failed: - print("-" * 60) - print(f"运行失败/无可读 check_golden.json 的样例 ({len(run_failed)}): {run_failed}") - for idx in run_failed: - print(f" {idx}: {LOGDIR / f'{idx}.log'}") - print("=" * 60) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/benchmark/longmemeval/stats_session_review.py b/benchmark/longmemeval/stats_session_review.py deleted file mode 100644 index 345a9dee..00000000 --- a/benchmark/longmemeval/stats_session_review.py +++ /dev/null @@ -1,248 +0,0 @@ -#!/usr/bin/env python3 -"""Summarise LongMemEval ``session_review.json`` artifacts. - -This script is for upstream health checks before running ``golden_check``. -Samples with retryable per-session failures should be rerun as a whole; samples -with non-retryable fallback reviews are reported separately. - -Examples: - python benchmark/longmemeval/stats_session_review.py - python benchmark/longmemeval/stats_session_review.py --list-failed - python benchmark/longmemeval/stats_session_review.py --list-fallback - python benchmark/longmemeval/stats_session_review.py --json -""" - -import argparse -import json -from pathlib import Path - -REPO = Path(__file__).resolve().parents[2] -DATA = REPO / "datasets" / "longmemeval" -LOGDIR = REPO / "logs" / "session_review" -OUTPUT_FILENAME = "session_review.json" - - -def parse_args() -> argparse.Namespace: - """Parse command-line arguments.""" - p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - p.add_argument("--list-failed", action="store_true", help="list samples with retryable failed per-session reviews") - p.add_argument("--list-fallback", action="store_true", help="list non-retryable fallback reviews") - p.add_argument("--list-missing", action="store_true", help="list samples missing session_review.json") - p.add_argument("--list-run-failed", action="store_true", help="list launched samples without a healthy output") - p.add_argument("--json", action="store_true", help="emit the summary as JSON") - return p.parse_args() - - -def sample_ids() -> list[str]: - """List all numeric sample IDs.""" - ids = [p.name for p in DATA.iterdir() if p.is_dir() and p.name.isdigit()] - return sorted(ids, key=int) - - -def pct(num: int, den: int) -> str: - """Format a percentage.""" - return f"{(100.0 * num / den):.1f}%" if den else "n/a" - - -def load_json(path: Path) -> dict: - """Load a JSON object, returning {} on any error.""" - try: - with path.open(encoding="utf-8") as f: - data = json.load(f) - return data if isinstance(data, dict) else {} - except (OSError, json.JSONDecodeError): - return {} - - -def logged_sample_ids() -> list[str]: - """List sample IDs that have a session_review runner log.""" - if not LOGDIR.exists(): - return [] - ids = [p.stem for p in LOGDIR.glob("*.log") if p.stem.isdigit()] - return sorted(ids, key=int) - - -def review_block(data: dict) -> dict: - """Return the review block when present.""" - review = data.get("review") if isinstance(data, dict) else None - return review if isinstance(review, dict) else {} - - -def failure_details(data: dict) -> list[dict]: - """Return retryable failed_reviews when present.""" - failed_reviews = review_block(data).get("failed_reviews") - if not isinstance(failed_reviews, list): - return [] - return [item for item in failed_reviews if isinstance(item, dict) and not item.get("fallback")] - - -def fallback_details(data: dict) -> list[dict]: - """Return non-retryable fallback review details when present.""" - review = review_block(data) - fallback_reviews = review.get("fallback_reviews") - if isinstance(fallback_reviews, list): - return [item for item in fallback_reviews if isinstance(item, dict)] - - failed_reviews = review.get("failed_reviews") - if isinstance(failed_reviews, list): - return [item for item in failed_reviews if isinstance(item, dict) and item.get("fallback")] - return [] - - -def failure_count(data: dict) -> int: - """Return retryable failed review count.""" - review = review_block(data) - raw = review.get("num_failed_reviews") - raw_fallback = review.get("num_fallback_reviews") - if isinstance(raw, int) and isinstance(raw_fallback, int): - return max(0, raw - raw_fallback) - return len(failure_details(data)) - - -def fallback_count(data: dict) -> int: - """Return non-retryable fallback review count.""" - review = review_block(data) - raw = review.get("num_fallback_reviews") - if isinstance(raw, int): - return raw - return len(fallback_details(data)) - - -def question_id(data: dict) -> str: - """Return query.question_id when present.""" - query = data.get("query") if isinstance(data, dict) else None - if not isinstance(query, dict): - return "" - return str(query.get("question_id") or "").strip() - - -def main() -> int: - """Main entry point.""" - args = parse_args() - ids = sample_ids() - total = len(ids) - - healthy, failed, fallback, missing, unreadable = [], [], [], [], [] - total_failed_sessions = 0 - total_fallback_sessions = 0 - failed_details_by_id: dict[str, list[dict]] = {} - fallback_details_by_id: dict[str, list[dict]] = {} - question_id_by_id: dict[str, str] = {} - - for idx in ids: - path = DATA / idx / OUTPUT_FILENAME - if not path.exists(): - missing.append(idx) - continue - data = load_json(path) - if not data: - unreadable.append(idx) - continue - question_id_by_id[idx] = question_id(data) - n_failed = failure_count(data) - n_fallback = fallback_count(data) - if n_failed: - failed.append(idx) - total_failed_sessions += n_failed - failed_details_by_id[idx] = failure_details(data) - if n_fallback: - fallback.append(idx) - total_fallback_sessions += n_fallback - fallback_details_by_id[idx] = fallback_details(data) - if not n_failed: - healthy.append(idx) - - launched = logged_sample_ids() - healthy_set = set(healthy) - run_failed = [idx for idx in launched if idx not in healthy_set] - - if args.json: - print( - json.dumps( - { - "total": total, - "healthy": len(healthy), - "failed_samples": failed, - "failed_sample_count": len(failed), - "failed_session_count": total_failed_sessions, - "fallback_samples": fallback, - "fallback_sample_count": len(fallback), - "fallback_session_count": total_fallback_sessions, - "missing": missing, - "unreadable": unreadable, - "launched": len(launched), - "run_failed_or_unhealthy": run_failed, - "failed_details": failed_details_by_id, - "fallback_details": fallback_details_by_id, - }, - ensure_ascii=False, - indent=2, - ), - ) - return 0 - - print("=" * 60) - print("LongMemEval session_review 统计") - print("=" * 60) - print(f"样例总数 : {total}") - print(f"可继续产出 : {len(healthy)} ({pct(len(healthy), total)})") - print(f"有可重试失败 : {len(failed)}") - print(f"可重试失败 session : {total_failed_sessions}") - print(f"有不可重试 fallback : {len(fallback)}") - print(f"fallback session : {total_fallback_sessions}") - print(f"缺少 session_review : {len(missing)}") - print(f"损坏/无法解析 : {len(unreadable)}") - print(f"已启动过 (有 log) : {len(launched)}") - print(f"运行失败/非健康产出 : {len(run_failed)}") - print("-" * 60) - print("有可重试 failed_reviews 的样例需要整体重跑：") - if failed: - print(" ".join(failed)) - print("重跑命令示例：") - print(f"python benchmark/longmemeval/run_session_review.py --start {failed[0]} --end {failed[0]}") - else: - print("(none)") - if fallback: - print("-" * 60) - print("不可重试 fallback 的样例不用重跑：") - for idx in fallback: - details = fallback_details_by_id.get(idx) or [] - session_ids = [str(item.get("session_id") or "(unknown)") for item in details] - qid = question_id_by_id.get(idx) - sample_label = f"{idx}({qid})" if qid else idx - print(f"{sample_label}: {' '.join(session_ids) if session_ids else '(unknown)'}") - - if args.list_failed and failed: - print("-" * 60) - for idx in failed: - details = failed_details_by_id.get(idx) or [] - print(f"{idx}: {DATA / idx / OUTPUT_FILENAME} failed_sessions={len(details)}") - for item in details: - session_id = item.get("session_id", "(unknown)") - error = str(item.get("error") or "").replace("\n", " ") - print(f" - {session_id}: {error}") - if args.list_fallback and fallback: - print("-" * 60) - for idx in fallback: - details = fallback_details_by_id.get(idx) or [] - print(f"{idx}: {DATA / idx / OUTPUT_FILENAME} fallback_sessions={len(details)}") - for item in details: - session_id = item.get("session_id", "(unknown)") - reason = str(item.get("fallback_reason") or "fallback") - error = str(item.get("error") or "").replace("\n", " ") - raw_saved = "yes" if item.get("raw_session") else "no" - print(f" - {session_id}: reason={reason} raw_session_saved={raw_saved} error={error}") - if args.list_missing and missing: - print("-" * 60) - print(f"缺少 session_review.json 的样例 ({len(missing)}): {missing}") - if args.list_run_failed and run_failed: - print("-" * 60) - print(f"运行失败/非健康产出的样例 ({len(run_failed)}): {run_failed}") - for idx in run_failed: - print(f" {idx}: {LOGDIR / f'{idx}.log'}") - print("=" * 60) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/benchmark/result-beam.md b/benchmark/result-beam.md new file mode 100644 index 00000000..4ec5b71a --- /dev/null +++ b/benchmark/result-beam.md @@ -0,0 +1,36 @@ +# beam result + +## longmemeval版本的prompt + +### 100K + + +| 题型 | Prompted(limit=15) | Agentic | +|---|---|---| +| abstention | 0.525 | 0.575 | +| contradiction_resolution | 0.100 | 0.384 | +| event_ordering | 0.403 | 0.465 | +| information_extraction | 0.618 | 0.884 | +| instruction_following | 0.481 | 0.719 | +| knowledge_update | 0.637 | 0.650 | +| multi_session_reasoning | 0.444 | 0.633 | +| preference_following | 0.706 | 0.829 | +| summarization | 0.423 | 0.617 | +| temporal_reasoning | 0.344 | 0.550 | +| **OVERALL** | **0.468** | **0.631** | + +### 1M + +| 题型 | Prompted(limit=15) | Prompted Binary | Agentic | Agentic Binary | +|---|---|---|---|---| +| abstention | 0.464 | 0.464 | 0.514 | 0.514 | +| contradiction_resolution | 0.079 | 0.068 | 0.373 | 0.339 | +| event_ordering | 0.455 | 0.334 | 0.547 | 0.450 | +| information_extraction | 0.653 | 0.589 | 0.818 | 0.764 | +| instruction_following | 0.541 | 0.524 | 0.765 | 0.745 | +| knowledge_update | 0.571 | 0.507 | 0.636 | 0.629 | +| multi_session_reasoning | 0.426 | 0.324 | 0.593 | 0.540 | +| preference_following | 0.718 | 0.676 | 0.838 | 0.824 | +| summarization | 0.516 | 0.303 | 0.661 | 0.478 | +| temporal_reasoning | 0.198 | 0.169 | 0.394 | 0.383 | +| **OVERALL** | **0.462** | **0.396** | **0.614** | **0.567** | \ No newline at end of file diff --git a/benchmark/result-longmemeval.md b/benchmark/result-longmemeval.md new file mode 100644 index 00000000..59671395 --- /dev/null +++ b/benchmark/result-longmemeval.md @@ -0,0 +1,92 @@ +# LongMemEval 数据集测试结果 + +## cleaned-s + +**basic settings** + +1. 使用修改后的auto-memory prompt，关闭auto-dream机制 +2. reme-memory中的全部session的时间一定早于question的时间 + +**results ** + +1. Agentic answer框架回答，每次最多调用5次search + +| Category | Total | Correct | Wrong | Accuracy | +|---|---|---|---|---| +| single-session-user | 70 | 66 | 4 | 94.3% | +| single-session-assistant | 56 | 52 | 4 | 92.9% | +| knowledge-update | 78 | 60 | 18 | 76.9% | +| multi-session | 133 | 93 | 40 | 69.9% | +| temporal-reasoning | 133 | 78 | 55 | 58.6% | +| single-session-preference | 30 | 8 | 22 | 26.7% | +| **Overall** | **500** | **357** | **143** | **71.4%** | + +2. prompted-based amswer，每次固定使用原始query召回10个fileChunk + +| Category | Total | Correct | Wrong | Accuracy | +|---|---|---|---|---| +| single-session-assistant | 56 | 56 | 0 | 100.0% | +| single-session-user | 70 | 67 | 3 | 95.7% | +| knowledge-update | 78 | 69 | 9 | 88.5% | +| multi-session | 133 | 99 | 34 | 74.4% | +| temporal-reasoning | 133 | 83 | 50 | 62.4% | +| single-session-preference | 30 | 16 | 14 | 53.3% | +| **Overall** | **500** | **390** | **110** | **78.0%** | + +3. golden session。 使用与prompt-based answer相似的方法，唯一区别是，输入的chunk是longMemEval提供的golden session。 + +| Category | Total | Correct | Wrong | Accuracy | +|---|---|---|---|---| +| single-session-assistant | 56 | 56 | 0 | 100.0% | +| single-session-user | 70 | 69 | 1 | 98.6% | +| knowledge-update | 78 | 74 | 4 | 94.9% | +| temporal-reasoning | 133 | 124 | 9 | 93.2% | +| multi-session | 133 | 117 | 16 | 88.0% | +| single-session-preference | 30 | 17 | 13 | 56.7% | +| **Overall** | **500** | **457** | **43** | **91.4%** | + +4. golden session + time filter. 和上面一个实验的区别是，输入的golden被过滤了一次，要求输入session的时间戳必须早于question的时间才行。 + +一共被过滤掉了75个session，44个question受到了影响。temperal reasoning类型受影响最大。有20个case不包含任何一个groundtruth session。 根据golden session回答正确并且golden session非空，一共有424个case。 + +| Category | Total | Correct | Wrong | Accuracy | +|---|---|---|---|---| +| knowledge-update | 78 | 75 | 3 | 96.2% | +| single-session-user | 70 | 67 | 3 | 95.7% | +| multi-session | 133 | 122 | 11 | 91.7% | +| single-session-assistant | 56 | 55 | 1 | 98.2% | +| temporal-reasoning | 133 | 91 | 42 | 68.4% | +| single-session-preference | 30 | 16 | 14 | 53.3% | +| **Overall** | **500** | **426** | **74** | **85.2%** | + +5. 关闭auto-memory机制，根据原始query一次性混合检索召回原始session，计算recall. + +| Category | Total | yes-judge | recall@5 / yes | recall@10 / yes | +|---|---|---|---|---| +| knowledge-update | 78 | 75 | 99.3% | 100% | +| single-session-user | 70 | 67 | 100% | 100% | +| multi-session | 133 | 122 | 91.8% | 95.8% | +| single-session-assistant | 56 | 55 | 100% | 100% | +| temporal-reasoning | 133 | 91 | 87.6% | 94.2% | +| single-session-preference | 30 | 16 | 100% | 100% | +| **Overall** | **500** | **426** | **87.6%** | **94.2%** | + + +## 最终groundtruth + +### agentic + prompted（最终GT，2026-07-16） + + +| Category | Total | Agentic | Prompted limit=15 | +|---|---|---|---| +| single-session-assistant | 56 | 56/56 (100.0%) | 54/56 (96.4%) | +| single-session-user | 70 | 66/70 (94.3%) | 62/70 (88.6%) | +| knowledge-update | 78 | 75/78 (96.2%) | 67/78 (85.9%) | +| temporal-reasoning | 133 | 122/133 (91.7%) | 117/133 (88.0%) | +| multi-session | 133 | 115/133 (86.5%) | 101/133 (75.9%) | +| single-session-preference | 30 | 21/30 (70.0%) | 10/30 (33.3%) | +| **Overall** | **500** | **455/500 (91.0%)** | **411/500 (82.2%)** | + +Prompted token 消耗：总 input 13,111,421 (平均 26,275/题)，总 output 313,370 (平均 628/题)。 +平均 sessions_ingested: 44.8，dreams_triggered: 0。 + diff --git a/reme/config/beam.yaml b/reme/config/beam.yaml new file mode 100644 index 00000000..87cda210 --- /dev/null +++ b/reme/config/beam.yaml @@ -0,0 +1,586 @@ +# BEAM benchmark config — based on longmemeval.yaml (split) +# All background/cron jobs are converted to base (manually callable). +# Use with: resolve_app_config(config="beam.yaml", ...) + +service: + backend: http + +jobs: + # ── BEAM agentic answer (ReAct agent + search tool) ── + agentic_answer: + backend: base + description: "BEAM agentic answer job (ReAct agent with search tool)" + watch_dirs: [] + watch_suffixes: [] + parameters: + type: object + properties: + query: + type: string + description: "The query to ask" + query_time: + type: string + description: "ISO timestamp representing the query time" + default: "" + required: + - query + steps: + - backend: beam_agentic_answer_step + agent_wrapper: bench + + # ── BEAM rubric-based LLM-as-Judge ── + answer_judge: + backend: base + description: "BEAM rubric-based LLM-as-Judge: evaluate response against rubric criteria" + watch_dirs: [] + watch_suffixes: [] + parameters: + type: object + properties: + llm_response: + type: string + description: "The model's response to evaluate" + rubric: + type: array + description: "List of rubric criteria to check" + items: + type: string + probing_question: + type: string + description: "The original probing question" + default: "" + question_type: + type: string + description: "BEAM question type (e.g. event_ordering)" + default: "" + required: + - llm_response + - rubric + steps: + - backend: beam_rubric_judge_step + agent_wrapper: judge + + # ── Manual index update (replaces index_update_loop background) ── + index_update: + backend: base + description: "Manually trigger incremental index update for watched dirs." + watch_dirs: [daily_dir, digest_dir, dialog_dir] + watch_suffixes: [md, jsonl] + parameters: + type: object + properties: {} + steps: + - backend: init_changes_step + monitor_type: file_store + monitor_name: default + dispatch_steps: [update_index_step] + + # ── Manual digest catalog update (replaces digest_watch_loop background) ── + digest_update: + backend: base + description: "Manually trigger digest catalog update." + watch_dirs: [daily_dir, digest_dir] + watch_suffixes: [md] + parameters: + type: object + properties: {} + steps: + - backend: init_changes_step + monitor_type: file_catalog + monitor_name: digest + dispatch_steps: + - backend: update_catalog_step + file_catalog: digest + - backend: log_changes_step + + # ── Auto dream (same as default.yaml auto_dream, base mode) ── + # auto_dream: + # backend: base + # description: "Auto-dream: scan today's day-index and daily notes, globally extract merged units/topics, integrate digest units, write interests.yaml, and persist the dream catalog." + # parameters: + # type: object + # properties: + # date: + # type: string + # description: "YYYY-MM-DD to scan; defaults to today in the dreamer's timezone" + # default: "" + # hint: + # type: string + # description: "caller guidance passed through to dream extract/integrate" + # default: "" + # scan_days: + # type: integer + # description: "number of recent daily directories to scan, ending at date" + # default: 2 + # max_units: + # type: integer + # description: "maximum number of extracted memory units" + # default: 5 + # topic_count: + # type: integer + # description: "maximum number of final daily interest topics" + # default: 3 + # topic_diversity_days: + # type: integer + # description: "number of previous interests.yaml days to avoid repeating" + # default: 7 + # steps: + # - backend: dream_extract_step + # file_catalog: dream + # topic_session_id: interests + # scan_days: 2 + # max_units: 5 + # - backend: dream_integrate_step + # - backend: dream_topics_step + # topic_count: 3 + # topic_diversity_days: 7 + # - backend: dream_finish_step + # file_catalog: dream + + # ── Auto memory (same as default.yaml) ── + auto_memory: + backend: base + description: "Auto-memory: record conversation facts into a daily note" + parameters: + type: object + properties: + messages: + type: array + description: "messages" + items: + type: object + session_id: + type: string + description: "source conversation session identifier" + default: "" + memory_hint: + type: string + description: "optional hint" + date: + type: string + description: "YYYY-MM-DD daily note date; empty = infer from message timestamps or today" + default: "" + required: + - messages + steps: + - backend: beam_auto_memory_step + + # ── Reindex (full rebuild) ── + reindex: + backend: base + description: "wipe the file store and rebuild it from the existing files" + watch_dirs: [daily_dir, digest_dir, resource_dir] + watch_suffixes: [md, jsonl] + parameters: + type: object + properties: {} + steps: + - backend: clear_store_step + - backend: init_changes_step + monitor_type: file_store + monitor_name: default + dispatch_steps: [update_index_step] + + # ── Search ── + # start_date: + # type: string + # description: "optional inclusive start date filter (YYYY-MM-DD); results earlier than this date are excluded" + # end_date: + # type: string + # description: "optional inclusive end date filter (YYYY-MM-DD); results later than this date are excluded" + + search: + backend: base + description: "Hybrid workspace search (vector + BM25, RRF-fused) with deduplication." + parameters: + type: object + properties: + query: + type: string + description: "search query" + start_date: + type: string + description: "optional inclusive start date filter (YYYY-MM-DD); results earlier than this date are excluded" + end_date: + type: string + description: "optional inclusive end date filter (YYYY-MM-DD); results later than this date are excluded" + # vector_weight: + # type: number + # description: >- + # Optional weight balancing vector similarity vs BM25 keyword matching in the + # RRF fusion. Recommended value is 0.7, which provides a good balance between + # semantic (vector) similarity and lexical (BM25) matching. Values close to 0 + # emphasize BM25 keyword matching, values close to 1 emphasize vector semantic + # similarity. + required: + - query + steps: + - backend: search_step + vector_weight: 0.7 + candidate_multiplier: 5.0 + expand_links: true + max_links_per_direction: 10 + + add_draft: + backend: base + description: "Append text to the current draft list." + parameters: + type: object + properties: + text: + type: string + description: "draft text to append" + required: + - text + steps: + - backend: add_draft_step + + read_all_draft: + backend: base + description: "Read all draft text previously appended in the current tool context." + parameters: + type: object + properties: { } + steps: + - backend: read_all_draft_step + + python_execute: + backend: base + description: "Execute Python code and return printed stdout." + parameters: + type: object + properties: + code: + type: string + description: "Python code to execute. Print the final result to stdout." + timeout: + type: number + description: "Execution timeout in seconds; defaults to 60." + required: + - code + steps: + - backend: python_execute_step + + # ── File I/O jobs (needed by auto_memory agent tools) ── + daily_list: + backend: base + description: "List notes under a single day." + parameters: + type: object + properties: + date: + type: string + description: "YYYY-MM-DD; empty = today" + default: "" + steps: + - backend: daily_list_step + + daily_reindex: + backend: base + description: "Rebuild the day-index page daily/.md." + parameters: + type: object + properties: + date: + type: string + description: "YYYY-MM-DD; empty = today" + default: "" + steps: + - backend: daily_reindex_step + + frontmatter_update: + backend: base + description: "Merge key-values into a file's frontmatter." + parameters: + type: object + properties: + path: + type: string + description: "workspace-relative path" + metadata: + type: object + description: "key-values to merge" + required: + - path + - metadata + steps: + - backend: frontmatter_update_step + + move: + backend: base + description: "Move / rename a workspace file." + parameters: + type: object + properties: + src_path: + type: string + description: "workspace-relative source" + dst_path: + type: string + description: "workspace-relative destination" + overwrite: + type: boolean + default: false + retarget: + type: boolean + default: true + required: + - src_path + - dst_path + steps: + - backend: move_step + + read: + backend: base + description: "Read a markdown file under the workspace." + parameters: + type: object + properties: + path: + type: string + description: "workspace-relative path; markdown only" + start_line: + type: integer + end_line: + type: integer + required: + - path + steps: + - backend: read_step + with_neighbors: false + max_neighbors_per_direction: 10 + + write: + backend: base + description: "Write a markdown file." + parameters: + type: object + properties: + path: + type: string + name: + type: string + description: + type: string + content: + type: string + metadata: + type: object + required: + - path + - name + - description + - content + steps: + - backend: write_step + + daily_write: + backend: base + description: "Write a daily markdown note." + parameters: + type: object + properties: + name: + type: string + description: + type: string + session_id: + type: string + content: + type: string + date: + type: string + default: "" + metadata: + type: object + required: + - name + - description + - session_id + - content + steps: + - backend: daily_write_step + + edit: + backend: base + description: "Find-and-replace in a markdown file." + parameters: + type: object + properties: + path: + type: string + old: + type: string + new: + type: string + default: "" + required: + - path + - old + - new + steps: + - backend: edit_step + + frontmatter_read: + backend: base + description: "Read a file's frontmatter as a dict." + parameters: + type: object + properties: + path: + type: string + required: + - path + steps: + - backend: frontmatter_read_step + + node_search: + backend: base + description: "Digest node recall." + parameters: + type: object + properties: + query: + type: string + limit: + type: integer + default: 20 + required: + - query + steps: + - backend: node_search_step + vector_weight: 0.7 + candidate_multiplier: 5.0 + +components: + tokenizer: + default: + backend: regex + + as_embedding: + default: + backend: ${EMBEDDING_BACKEND:-openai} + model: ${EMBEDDING_MODEL_NAME:-text-embedding-v4} + credential: + api_key: ${EMBEDDING_API_KEY:-} + base_url: ${EMBEDDING_BASE_URL:-https://dashscope.aliyuncs.com/compatible-mode/v1} + dimensions: 1024 + + embedding_store: + default: + backend: local + as_embedding: default + + as_llm: + default: + backend: ${LLM_BACKEND:-openai} + model: ${LLM_MODEL_NAME:-qwen3.6-flash} + stream: true + context_size: 200000 + max_retries: 5 + retry_delay: 5.0 + credential: + api_key: ${LLM_API_KEY:-} + base_url: ${LLM_BASE_URL:-} + parameters: + max_tokens: 65536 + thinking_enable: false + judge: + backend: ${LLM_BACKEND:-openai} + model: ${JUDGE_MODEL_NAME:-qwen3.7-max} + stream: false + context_size: 200000 + max_retries: 5 + retry_delay: 5.0 + credential: + api_key: ${LLM_API_KEY:-} + base_url: ${LLM_BASE_URL:-} + parameters: + max_tokens: 65536 + thinking_enable: false + bench: + backend: ${LLM_BACKEND:-openai} + model: ${BENCH_MODEL_NAME:-qwen3.7-max} + stream: true + context_size: 400000 + max_retries: 5 + retry_delay: 5.0 + credential: + api_key: ${LLM_API_KEY:-} + base_url: ${LLM_BASE_URL:-} + parameters: + max_tokens: 65536 + thinking_enable: true + + agent_wrapper: + default: + backend: agentscope + as_llm: default + permission_mode: bypass + react_config: + max_iters: 30 + context_config: + trigger_ratio: 0.8 + reserve_ratio: 0.1 + tool_result_limit: 50000 + model_config: + max_retries: 1 + judge: + backend: agentscope + as_llm: judge + permission_mode: bypass + react_config: + max_iters: 1 + context_config: + trigger_ratio: 0.8 + reserve_ratio: 0.1 + tool_result_limit: 50000 + model_config: + max_retries: 1 + bench: + backend: agentscope + as_llm: bench + permission_mode: bypass + react_config: + max_iters: 30 + context_config: + trigger_ratio: 0.8 + reserve_ratio: 0.1 + tool_result_limit: 50000 + model_config: + max_retries: 1 + + file_graph: + default: + backend: local + + file_catalog: + default: + backend: local + resource: + backend: local + digest: + backend: local + dream: + backend: local + + file_chunker: + markdown: + backend: markdown + supported_extensions: ["md"] + default: + backend: default + supported_extensions: ["jsonl"] + + keyword_index: + default: + backend: bm25 + tokenizer: default + + file_store: + default: + backend: local + store_name: local + embedding_store: default + keyword_index: default + file_graph: default diff --git a/reme/config/jinli_lme.yaml b/reme/config/jinli_lme.yaml deleted file mode 100644 index 5b4a6f41..00000000 --- a/reme/config/jinli_lme.yaml +++ /dev/null @@ -1,419 +0,0 @@ -service: - backend: cli - -workspace_dir: ${LME_WORKSPACE_DIR:-datasets/longmemeval/1} -session_dir: history_session -resource_dir: session -daily_dir: daily -digest_dir: "" - -jobs: - auto_memory: - backend: base - description: "Extract every raw session into a search-friendly daily note (one note per session)." - parameters: - type: object - properties: { } - steps: - - backend: clear_paths_step # wipe old daily notes so this is a clean rebuild - config_keys: [daily_dir] - - backend: lme_auto_memory_step - agent_wrapper: lme_memory - - update_index: - backend: base - watch_dirs: [daily_dir] - watch_suffixes: [md] - steps: - - backend: clear_store_step - - backend: init_changes_step - monitor_type: file_store - monitor_name: default - dispatch_steps: [update_index_step] - - extract_session_by_id: - backend: base - description: "Given a session_id shown in a search result, go back to the original raw session and extract everything in it that is relevant to the current question. Use this when a search hit looks relevant but the distilled note lacks an exact number, date, or wording." - parameters: - type: object - properties: - session_id: - type: string - description: "the session_id from a search result header" - required: - - session_id - steps: - - backend: lme_extract_session_step - agent_wrapper: lme_extract - - vector_search: - backend: base - description: "Dense semantic search over the memory notes. Returns each hit with its source note path and session_id." - parameters: - type: object - properties: - query: - type: string - description: "search query" - required: - - query - steps: - - backend: vector_search_step - include_source: true - - bm25_search: - backend: base - description: "Keyword (BM25) search over the memory notes. Returns each hit with its source note path and session_id." - parameters: - type: object - properties: - query: - type: string - description: "search query" - required: - - query - steps: - - backend: bm25_search_step - include_source: true - - agentic_answer: - backend: base - description: "Answer query.json from indexed memory using vector/bm25 search + session extraction." - parameters: - type: object - properties: { } - steps: - - backend: clear_paths_step # drop the previous answer so each run rewrites cleanly - paths: [mem_answer.json] - - backend: lme_agentic_answer_step - agent_wrapper: lme_agentic_answer - - llm_judge: - backend: base - description: "Judge mem_answer.json against answer.json and write the judgement back into mem_answer.json." - parameters: - type: object - properties: { } - steps: - - backend: lme_llm_judge_step - agent_wrapper: lme_judge - - version: - backend: base - description: "return reme package version" - parameters: - type: object - properties: { } - steps: - - backend: version_step - - search: - backend: base - description: "Hybrid workspace search (vector + BM25, RRF-fused)." - parameters: - type: object - properties: - query: - type: string - description: "search query" - start_date: - type: string - description: "optional inclusive start date filter (YYYY-MM-DD); results earlier than this date are excluded" - end_date: - type: string - description: "optional inclusive end date filter (YYYY-MM-DD); results later than this date are excluded" - required: - - query - steps: - - backend: search_step - vector_weight: 0.7 - candidate_multiplier: 5.0 - expand_links: true - max_links_per_direction: 10 - - add_draft: - backend: base - description: "Append text to the current draft list." - parameters: - type: object - properties: - text: - type: string - description: "draft text to append" - required: - - text - steps: - - backend: add_draft_step - - read_all_draft: - backend: base - description: "Read all draft text previously appended in the current tool context." - parameters: - type: object - properties: { } - steps: - - backend: read_all_draft_step - - python_execute: - backend: base - description: "Execute Python code and return printed stdout." - parameters: - type: object - properties: - code: - type: string - description: "Python code to execute. Print the final result to stdout." - timeout: - type: number - description: "Execution timeout in seconds; defaults to 60." - required: - - code - steps: - - backend: python_execute_step - - session_review: - backend: base - description: "Review every session for query/answer-relevant evidence and write session_review.json." - parameters: - type: object - properties: { } - steps: - - backend: clear_paths_step # drop the previous review so each run rewrites cleanly - paths: [session_review.json] - - backend: lme_session_review_step - agent_wrapper: lme_review - - golden_check: - backend: base - description: "Read session_review.json and judge whether the golden answer is reasonable." - parameters: - type: object - properties: { } - steps: - - backend: clear_paths_step # drop the previous verdict so each run rewrites cleanly - paths: [check_golden.json] - - backend: wait_for_paths_step # wait until session_review finishes; comment this step to fail fast instead - paths: [session_review.json] - poll_seconds: 5 - log_every_seconds: 60 - - backend: lme_golden_check_step - agent_wrapper: lme_judge - - final_answer_review: - backend: base - description: "Review one LongMemEval golden answer from all sessions available by question_date." - parameters: - type: object - properties: { } - steps: - - backend: lme_final_answer_review_step - agent_wrapper: lme_final_answer_review - reference_paths: - - benchmark/longmemeval/golden_check_list_false.jsonl - - benchmark/longmemeval/merge_confirm_jinli_false.jsonl - retry_initial_seconds: 5 - retry_max_seconds: 300 - -components: - tokenizer: - default: - backend: regex - - as_embedding: - default: - backend: ${EMBEDDING_BACKEND:-openai} - model: ${EMBEDDING_MODEL_NAME:-text-embedding-v4} - dimensions: 1024 - credential: - api_key: ${EMBEDDING_API_KEY:-} - base_url: ${EMBEDDING_BASE_URL:-https://dashscope.aliyuncs.com/compatible-mode/v1} - parameters: { } - - embedding_store: - default: - backend: local - as_embedding: default - - as_llm: - default: - backend: ${LLM_BACKEND:-openai} - model: ${LLM_MODEL_NAME:-qwen3.7-max} - stream: true - context_size: 1000000 - max_retries: 3 - credential: - api_key: ${LLM_API_KEY:-} - base_url: ${LLM_BASE_URL:-} - parameters: - max_tokens: 65536 - - plus: - backend: ${LLM_BACKEND:-openai} - model: qwen3.7-plus - stream: true - context_size: 1000000 - max_retries: 3 - credential: - api_key: ${LLM_API_KEY:-} - base_url: ${LLM_BASE_URL:-} - parameters: - max_tokens: 65536 - - agent_wrapper: - default: - backend: agentscope - as_llm: default - permission_mode: bypass - react_config: - max_iters: 30 - context_config: - trigger_ratio: 0.89 - reserve_ratio: 0.1 - tool_result_limit: 50000 - model_config: - max_retries: 3 - - agentic_search_agentwrapper: - backend: agentscope - as_llm: default - cwd: session - permission_mode: bypass - builtin_tools: false - job_tools: - - search - - add_draft - - read_all_draft - - python_execute - sequential_tool_calls: true - react_config: - max_iters: 100 - context_config: - trigger_ratio: 0.89 - reserve_ratio: 0.1 - tool_result_limit: 1000000 - model_config: - max_retries: 3 - - lme_memory: - backend: agentscope - as_llm: plus - permission_mode: bypass - builtin_tools: false - react_config: - max_iters: 8 - context_config: - trigger_ratio: 0.89 - reserve_ratio: 0.1 - tool_result_limit: 1000000 - model_config: - max_retries: 3 - - lme_extract: - backend: agentscope - as_llm: plus - permission_mode: bypass - builtin_tools: false - react_config: - max_iters: 3 - context_config: - trigger_ratio: 0.89 - reserve_ratio: 0.1 - tool_result_limit: 1000000 - model_config: - max_retries: 3 - - lme_agentic_answer: - backend: agentscope - as_llm: default - cwd: session - permission_mode: bypass - builtin_tools: false - job_tools: - - vector_search - - bm25_search - - python_execute - - extract_session_by_id - sequential_tool_calls: true - react_config: - max_iters: 40 - context_config: - trigger_ratio: 0.89 - reserve_ratio: 0.1 - tool_result_limit: 1000000 - model_config: - max_retries: 3 - - claude_code: - backend: claude_code - model: ${CLAUDE_CODE_MODEL_NAME:-glm-5.2} - api_key: ${CLAUDE_CODE_API_KEY:-} - base_url: ${CLAUDE_CODE_BASE_URL:-https://dashscope.aliyuncs.com/apps/anthropic} - permission_mode: bypassPermissions - - lme_final_answer_review: - backend: claude_code - model: ${CLAUDE_CODE_MODEL_NAME:-claude-opus-4-8} - api_key: ${CLAUDE_CODE_API_KEY:-} - base_url: ${CLAUDE_CODE_BASE_URL:-} - cwd: session - permission_mode: bypassPermissions - - lme_review: - backend: agentscope - as_llm: plus - permission_mode: bypass - builtin_tools: false - react_config: - max_iters: 10 - context_config: - trigger_ratio: 0.89 - reserve_ratio: 0.1 - tool_result_limit: 1000000 - model_config: - max_retries: 3 - - lme_judge: - backend: agentscope - as_llm: default - permission_mode: bypass - builtin_tools: false - job_tools: - - python_execute - sequential_tool_calls: true - react_config: - max_iters: 50 - context_config: - trigger_ratio: 0.89 - reserve_ratio: 0.1 - tool_result_limit: 1000000 - model_config: - max_retries: 3 - - file_graph: - default: - backend: local - - file_chunker: - markdown: - backend: markdown - supported_extensions: [ "md" ] - embed_toc: true - max_ast_sections: 100 - default: - backend: default - supported_extensions: [ "json", "jsonl" ] - chunk_byte_size: 100000 - - keyword_index: - default: - backend: bm25 - tokenizer: default - - file_store: - default: - backend: local - store_name: local - embedding_store: default -# embedding_store: "" - keyword_index: default - file_graph: default diff --git a/reme/config/lme.yaml b/reme/config/lme.yaml new file mode 100644 index 00000000..e32d2037 --- /dev/null +++ b/reme/config/lme.yaml @@ -0,0 +1,583 @@ +# LongMemEval benchmark config — based on longmemeval.yaml (split) +# All background/cron jobs are converted to base (manually callable). +# Use with: resolve_app_config(config="lme.yaml", ...) + +service: + backend: http + +jobs: + # ── LongMemEval agentic answer (ReAct agent + search tool) ── + agentic_answer: + backend: base + description: "LongMemEval agentic answer job (ReAct agent with search tool)" + watch_dirs: [] + watch_suffixes: [] + parameters: + type: object + properties: + query: + type: string + description: "The query to ask" + query_time: + type: string + description: "ISO timestamp representing the query time" + default: "" + required: + - query + steps: + - backend: lme_agentic_answer_step + agent_wrapper: bench + + # ── LLM-as-Judge for evaluating answers ── + answer_judge: + backend: base + description: "LLM-as-Judge: evaluate agent answer against golden answer" + watch_dirs: [] + watch_suffixes: [] + parameters: + type: object + properties: + query: + type: string + description: "The question being asked" + agent_answer: + type: string + description: "The model's answer to evaluate" + golden_answer: + type: string + description: "The correct/golden answer" + question_type: + type: string + description: "The question type for prompt selection" + default: "" + required: + - query + - agent_answer + - golden_answer + steps: + - backend: lme_answer_judge_step + agent_wrapper: judge + + # ── Manual index update (replaces index_update_loop background) ── + index_update: + backend: base + description: "Manually trigger incremental index update for watched dirs." + watch_dirs: [daily_dir, digest_dir, dialog_dir] + watch_suffixes: [md, jsonl] + parameters: + type: object + properties: {} + steps: + - backend: init_changes_step + monitor_type: file_store + monitor_name: default + dispatch_steps: [update_index_step] + + # ── Manual digest catalog update (replaces digest_watch_loop background) ── + digest_update: + backend: base + description: "Manually trigger digest catalog update." + watch_dirs: [daily_dir, digest_dir] + watch_suffixes: [md] + parameters: + type: object + properties: {} + steps: + - backend: init_changes_step + monitor_type: file_catalog + monitor_name: digest + dispatch_steps: + - backend: update_catalog_step + file_catalog: digest + - backend: log_changes_step + + # ── Auto dream (same as default.yaml auto_dream, base mode) ── + # auto_dream: + # backend: base + # description: "Auto-dream: scan today's day-index and daily notes, globally extract merged units/topics, integrate digest units, write interests.yaml, and persist the dream catalog." + # parameters: + # type: object + # properties: + # date: + # type: string + # description: "YYYY-MM-DD to scan; defaults to today in the dreamer's timezone" + # default: "" + # hint: + # type: string + # description: "caller guidance passed through to dream extract/integrate" + # default: "" + # scan_days: + # type: integer + # description: "number of recent daily directories to scan, ending at date" + # default: 2 + # max_units: + # type: integer + # description: "maximum number of extracted memory units" + # default: 5 + # topic_count: + # type: integer + # description: "maximum number of final daily interest topics" + # default: 3 + # topic_diversity_days: + # type: integer + # description: "number of previous interests.yaml days to avoid repeating" + # default: 7 + # steps: + # - backend: dream_extract_step + # file_catalog: dream + # topic_session_id: interests + # scan_days: 2 + # max_units: 5 + # - backend: dream_integrate_step + # - backend: dream_topics_step + # topic_count: 3 + # topic_diversity_days: 7 + # - backend: dream_finish_step + # file_catalog: dream + + # ── Auto memory (same as default.yaml) ── + auto_memory: + backend: base + description: "Auto-memory: record conversation facts into a daily note" + parameters: + type: object + properties: + messages: + type: array + description: "messages" + items: + type: object + session_id: + type: string + description: "source conversation session identifier" + default: "" + memory_hint: + type: string + description: "optional hint" + date: + type: string + description: "YYYY-MM-DD daily note date; empty = infer from message timestamps or today" + default: "" + required: + - messages + steps: + - backend: lme_auto_memory_step + + # ── Reindex (full rebuild) ── + reindex: + backend: base + description: "wipe the file store and rebuild it from the existing files" + watch_dirs: [daily_dir, digest_dir, resource_dir] + watch_suffixes: [md, jsonl] + parameters: + type: object + properties: {} + steps: + - backend: clear_store_step + - backend: init_changes_step + monitor_type: file_store + monitor_name: default + dispatch_steps: [update_index_step] + + # ── Search ── + # start_date: + # type: string + # description: "optional inclusive start date filter (YYYY-MM-DD); results earlier than this date are excluded" + # end_date: + # type: string + # description: "optional inclusive end date filter (YYYY-MM-DD); results later than this date are excluded" + + search: + backend: base + description: "Hybrid workspace search (vector + BM25, RRF-fused) with deduplication." + parameters: + type: object + properties: + query: + type: string + description: "search query" + start_date: + type: string + description: "optional inclusive start date filter (YYYY-MM-DD); results earlier than this date are excluded" + end_date: + type: string + description: "optional inclusive end date filter (YYYY-MM-DD); results later than this date are excluded" + # vector_weight: + # type: number + # description: >- + # Optional weight balancing vector similarity vs BM25 keyword matching in the + # RRF fusion. Recommended value is 0.7, which provides a good balance between + # semantic (vector) similarity and lexical (BM25) matching. Values close to 0 + # emphasize BM25 keyword matching, values close to 1 emphasize vector semantic + # similarity. + required: + - query + steps: + - backend: search_step + vector_weight: 0.7 + candidate_multiplier: 5.0 + expand_links: true + max_links_per_direction: 10 + + add_draft: + backend: base + description: "Append text to the current draft list." + parameters: + type: object + properties: + text: + type: string + description: "draft text to append" + required: + - text + steps: + - backend: add_draft_step + + read_all_draft: + backend: base + description: "Read all draft text previously appended in the current tool context." + parameters: + type: object + properties: { } + steps: + - backend: read_all_draft_step + + python_execute: + backend: base + description: "Execute Python code and return printed stdout." + parameters: + type: object + properties: + code: + type: string + description: "Python code to execute. Print the final result to stdout." + timeout: + type: number + description: "Execution timeout in seconds; defaults to 60." + required: + - code + steps: + - backend: python_execute_step + + # ── File I/O jobs (needed by auto_memory agent tools) ── + daily_list: + backend: base + description: "List notes under a single day." + parameters: + type: object + properties: + date: + type: string + description: "YYYY-MM-DD; empty = today" + default: "" + steps: + - backend: daily_list_step + + daily_reindex: + backend: base + description: "Rebuild the day-index page daily/.md." + parameters: + type: object + properties: + date: + type: string + description: "YYYY-MM-DD; empty = today" + default: "" + steps: + - backend: daily_reindex_step + + frontmatter_update: + backend: base + description: "Merge key-values into a file's frontmatter." + parameters: + type: object + properties: + path: + type: string + description: "workspace-relative path" + metadata: + type: object + description: "key-values to merge" + required: + - path + - metadata + steps: + - backend: frontmatter_update_step + + move: + backend: base + description: "Move / rename a workspace file." + parameters: + type: object + properties: + src_path: + type: string + description: "workspace-relative source" + dst_path: + type: string + description: "workspace-relative destination" + overwrite: + type: boolean + default: false + retarget: + type: boolean + default: true + required: + - src_path + - dst_path + steps: + - backend: move_step + + read: + backend: base + description: "Read a markdown file under the workspace." + parameters: + type: object + properties: + path: + type: string + description: "workspace-relative path; markdown only" + start_line: + type: integer + end_line: + type: integer + required: + - path + steps: + - backend: read_step + with_neighbors: false + max_neighbors_per_direction: 10 + + write: + backend: base + description: "Write a markdown file." + parameters: + type: object + properties: + path: + type: string + name: + type: string + description: + type: string + content: + type: string + metadata: + type: object + required: + - path + - name + - description + - content + steps: + - backend: write_step + + daily_write: + backend: base + description: "Write a daily markdown note." + parameters: + type: object + properties: + name: + type: string + description: + type: string + session_id: + type: string + content: + type: string + date: + type: string + default: "" + metadata: + type: object + required: + - name + - description + - session_id + - content + steps: + - backend: daily_write_step + + edit: + backend: base + description: "Find-and-replace in a markdown file." + parameters: + type: object + properties: + path: + type: string + old: + type: string + new: + type: string + default: "" + required: + - path + - old + - new + steps: + - backend: edit_step + + frontmatter_read: + backend: base + description: "Read a file's frontmatter as a dict." + parameters: + type: object + properties: + path: + type: string + required: + - path + steps: + - backend: frontmatter_read_step + + node_search: + backend: base + description: "Digest node recall." + parameters: + type: object + properties: + query: + type: string + limit: + type: integer + default: 20 + required: + - query + steps: + - backend: node_search_step + vector_weight: 0.7 + candidate_multiplier: 5.0 + +components: + tokenizer: + default: + backend: regex + + as_embedding: + default: + backend: ${EMBEDDING_BACKEND:-openai} + model: ${EMBEDDING_MODEL_NAME:-text-embedding-v4} + credential: + api_key: ${EMBEDDING_API_KEY:-} + base_url: ${EMBEDDING_BASE_URL:-https://dashscope.aliyuncs.com/compatible-mode/v1} + dimensions: 1024 + + embedding_store: + default: + backend: local + as_embedding: default + + as_llm: + default: + backend: ${LLM_BACKEND:-openai} + model: ${LLM_MODEL_NAME:-qwen3.6-flash} + stream: true + context_size: 200000 + max_retries: 3 + retry_delay: 5.0 + credential: + api_key: ${LLM_API_KEY:-} + base_url: ${LLM_BASE_URL:-} + parameters: + max_tokens: 65536 + thinking_enable: false + judge: + backend: ${LLM_BACKEND:-openai} + model: ${JUDGE_MODEL_NAME:-qwen3.7-max} + stream: false + context_size: 200000 + max_retries: 5 + credential: + api_key: ${LLM_API_KEY:-} + base_url: ${LLM_BASE_URL:-} + parameters: + max_tokens: 65536 + thinking_enable: false + bench: + backend: ${LLM_BACKEND:-openai} + model: ${BENCH_MODEL_NAME:-qwen3.7-max} + stream: true + context_size: 400000 + max_retries: 5 + retry_delay: 5.0 + credential: + api_key: ${LLM_API_KEY:-} + base_url: ${LLM_BASE_URL:-} + parameters: + max_tokens: 65536 + thinking_enable: true + + agent_wrapper: + default: + backend: agentscope + as_llm: default + permission_mode: bypass + react_config: + max_iters: 30 + context_config: + trigger_ratio: 0.8 + reserve_ratio: 0.1 + tool_result_limit: 50000 + model_config: + max_retries: 1 + judge: + backend: agentscope + as_llm: judge + permission_mode: bypass + react_config: + max_iters: 1 + context_config: + trigger_ratio: 0.8 + reserve_ratio: 0.1 + tool_result_limit: 50000 + model_config: + max_retries: 1 + bench: + backend: agentscope + as_llm: bench + permission_mode: bypass + react_config: + max_iters: 30 + context_config: + trigger_ratio: 0.8 + reserve_ratio: 0.1 + tool_result_limit: 50000 + model_config: + max_retries: 1 + + file_graph: + default: + backend: local + + file_catalog: + default: + backend: local + resource: + backend: local + digest: + backend: local + dream: + backend: local + + file_chunker: + markdown: + backend: markdown + supported_extensions: ["md"] + default: + backend: default + supported_extensions: ["jsonl"] + + keyword_index: + default: + backend: bm25 + tokenizer: default + + file_store: + default: + backend: local + store_name: local + embedding_store: default + keyword_index: default + file_graph: default diff --git a/reme/schema/application_config.py b/reme/schema/application_config.py index 2454e6b9..00a0b286 100644 --- a/reme/schema/application_config.py +++ b/reme/schema/application_config.py @@ -39,6 +39,7 @@ class ApplicationConfig(BaseModel): resource_dir: str = Field(default="resource", description="Subdirectory for external assets") daily_dir: str = Field(default="daily", description="Subdirectory for daily memory") digest_dir: str = Field(default="digest", description="Subdirectory for digest memory") + dialog_dir: str = Field(default="session/dialog", description="Subdirectory for dialog session transcripts") enable_logo: bool = Field(default=True, description="Show ASCII logo on startup") timezone: str | None = Field(default="Asia/Shanghai", description="IANA timezone; None uses local time") language: str = Field(default="", description="Default language for LLM interactions") diff --git a/reme/steps/benchmark/__init__.py b/reme/steps/benchmark/__init__.py index cab5b241..a5fdc8aa 100644 --- a/reme/steps/benchmark/__init__.py +++ b/reme/steps/benchmark/__init__.py @@ -1,12 +1,17 @@ """Benchmark steps.""" -from . import lme -from .lme import ContextAnswerStep, GoldenCheckStep, LmeLlmJudgeStep, SessionReviewStep +from . import base, lme, beam +from .base import BaseAgenticAnswerStep +from .lme import LmeAgenticAnswerStep, LmeAnswerJudgeStep +from .beam import BeamAgenticAnswerStep, BeamRubricJudgeStep __all__ = [ - "ContextAnswerStep", - "GoldenCheckStep", - "LmeLlmJudgeStep", - "SessionReviewStep", + "BaseAgenticAnswerStep", + "LmeAgenticAnswerStep", + "LmeAnswerJudgeStep", + "BeamAgenticAnswerStep", + "BeamRubricJudgeStep", + "base", "lme", + "beam", ] diff --git a/reme/steps/benchmark/base/__init__.py b/reme/steps/benchmark/base/__init__.py new file mode 100644 index 00000000..2b31bb0d --- /dev/null +++ b/reme/steps/benchmark/base/__init__.py @@ -0,0 +1,7 @@ +"""Shared base classes for benchmark steps.""" + +from .agentic_answer import BaseAgenticAnswerStep + +__all__ = [ + "BaseAgenticAnswerStep", +] diff --git a/reme/steps/benchmark/base/agentic_answer.py b/reme/steps/benchmark/base/agentic_answer.py new file mode 100644 index 00000000..7eb45af1 --- /dev/null +++ b/reme/steps/benchmark/base/agentic_answer.py @@ -0,0 +1,102 @@ +"""Shared base class for benchmark agentic-answer steps.""" + +import os +import threading + +from ...base_step import BaseStep +from ....enumeration import ChunkEnum + +# --------------------------------------------------------------------------- +# Process-safe & thread-safe counter for unique tool_context_id. +# PID guarantees cross-process uniqueness (multiprocessing Pool); +# threading.Lock + monotonic counter guarantees thread safety within a process. +# --------------------------------------------------------------------------- +_TOOL_CTX_LOCK = threading.Lock() +_TOOL_CTX_SEQ = 0 + + +def _next_tool_context_id(prefix: str) -> str: + global _TOOL_CTX_SEQ + with _TOOL_CTX_LOCK: + _TOOL_CTX_SEQ += 1 + seq = _TOOL_CTX_SEQ + return f"{prefix}_{os.getpid()}_{seq}" + + +class BaseAgenticAnswerStep(BaseStep): + """Base ReAct-agent answer step shared by BEAM and LongMemEval benchmarks. + + Subclasses only need to set: + TOOL_CONTEXT_PREFIX (str): prefix used to build the unique tool_context_id. + + And apply their own ``@R.register(...)`` decorator and docstring. + + Inputs (from RuntimeContext): + query (str, required): The question to answer. + query_time (str, optional): ISO timestamp representing the query time, + used to ground the agent's temporal context. + + Output (written to context.response.answer): + The agent's final answer text. + """ + + MAX_ITERATION = 10 + TOOL_CONTEXT_PREFIX: str = "agentic_answer" + + async def execute(self): + assert self.context is not None + query: str = self.context.get("query", "") + query_time: str | None = self.context.get("query_time") + + if not query: + self.context.response.success = False + self.context.response.answer = "Skipped: empty query" + return self.context.response + + # Build system prompt with optional temporal context + sys_prompt = self.get_prompt("system_prompt") + if query_time: + sys_prompt += "\n" + self.prompt_format("temporal_hint", query_time=query_time) + + wrapper_kwargs = { + "system_prompt": sys_prompt, + "job_tools": ["search", "add_draft", "read_all_draft"], + "react_config": {"max_iters": self.MAX_ITERATION}, + "tool_context_id": _next_tool_context_id(self.TOOL_CONTEXT_PREFIX), + } + + if self.context.stream: + text = await self._stream_reply(query, **wrapper_kwargs) + else: + result = await self.agent_wrapper.reply(query, **wrapper_kwargs) + text = (result.get("result") or "").strip() + + self.logger.debug(f"[{self.name}] response: {text!r}") + + self.context.response.success = True + self.context.response.answer = text + self.context.response.metadata.update( + { + "query": query, + "query_time": query_time, + "sys_prompt": sys_prompt, + "response": text, + }, + ) + return self.context.response + + async def _stream_reply(self, query: str, **wrapper_kwargs) -> str: + """Stream unified chunks to the context stream queue.""" + assert self.context is not None + text_parts: list[str] = [] + + async for chunk in self.agent_wrapper.reply_stream(query, **wrapper_kwargs): + await self.context.add_stream_string(chunk.chunk, chunk.chunk_type) + + if chunk.chunk_type == ChunkEnum.CONTENT and isinstance(chunk.chunk, str): + text_parts.append(chunk.chunk) + + if chunk.session_id: + self.context.response.metadata["session_id"] = chunk.session_id + + return "".join(text_parts).strip() diff --git a/reme/steps/benchmark/beam/__init__.py b/reme/steps/benchmark/beam/__init__.py new file mode 100644 index 00000000..e8c6e48f --- /dev/null +++ b/reme/steps/benchmark/beam/__init__.py @@ -0,0 +1,11 @@ +"""BEAM benchmark steps.""" + +from .agentic_answer import BeamAgenticAnswerStep +from .llm_judge import BeamRubricJudgeStep +from .auto_memory import BeamAutoMemoryStep + +__all__ = [ + "BeamAgenticAnswerStep", + "BeamRubricJudgeStep", + "BeamAutoMemoryStep", +] diff --git a/reme/steps/benchmark/beam/agentic_answer.py b/reme/steps/benchmark/beam/agentic_answer.py new file mode 100644 index 00000000..3b9a02fc --- /dev/null +++ b/reme/steps/benchmark/beam/agentic_answer.py @@ -0,0 +1,16 @@ +"""BEAM agentic answer step – ReAct agent that answers questions using the search tool.""" + +from ....components import R +from ..base import BaseAgenticAnswerStep + + +@R.register("beam_agentic_answer_step") +class BeamAgenticAnswerStep(BaseAgenticAnswerStep): + """Answer a BEAM probing question via ReAct agent with access to the search tool. + + The agent uses the ``agent_wrapper`` component in ReAct mode, calling the + ``search`` job tool to retrieve relevant memory chunks before generating + a final answer. + """ + + TOOL_CONTEXT_PREFIX = "beam_agentic_answer" diff --git a/reme/steps/benchmark/beam/agentic_answer.yaml b/reme/steps/benchmark/beam/agentic_answer.yaml new file mode 100644 index 00000000..ab4f00aa --- /dev/null +++ b/reme/steps/benchmark/beam/agentic_answer.yaml @@ -0,0 +1,15 @@ +system_prompt: | + You are a memory retrieval assistant. You MUST use the search tool to find information before answering. + - Your total time of tool calls should be at most 9 times + + ## Search Strategy + - You can call 'search' tool to search multiple times (at least once) with different queries to gather comprehensive information. + ## Draft Tool + - Use 'add_draft' to save key findings during search, and 'read_all_draft' to review all saved notes before answering. + ## Answer Rules + - Answer based ONLY on retrieved context. + - Output ONLY the direct factual answer — no reasoning, no search process, no elaboration. + - If information is not founded or not sufficient after multiple searches, reply: 'Information not found.' + +temporal_hint: | + Current time context: {query_time} diff --git a/reme/steps/benchmark/beam/auto_memory.py b/reme/steps/benchmark/beam/auto_memory.py new file mode 100644 index 00000000..1a2a9945 --- /dev/null +++ b/reme/steps/benchmark/beam/auto_memory.py @@ -0,0 +1,132 @@ +"""auto_memory (beam) — AutoMemoryStep with timestamp interpolation and daily_write date default.""" + +from datetime import datetime, timedelta + +from agentscope.message import Msg + +from ...evolve.auto_memory import AutoMemoryStep, _normalize_msg_timestamp +from ....components import R + + +def _parse_iso_seconds(value: str) -> datetime | None: + """Parse an ISO-8601 timestamp that is precise to at least seconds. + + Accepts formats like: + 2026-07-01T14:30:00 + 2026-07-01T14:30:00Z + 2026-07-01T14:30:00+08:00 + 2026-07-01T14:30:00.123456 + + Rejects date-only (``2026-07-01``) or minute-only (``2026-07-01T14:30``). + Returns ``None`` when the value does not satisfy the requirements. + """ + text = str(value).strip() + # Minimum valid: YYYY-MM-DDTHH:MM:SS = 19 chars + if len(text) < 19: + return None + # Must contain 'T' separator and at least HH:MM:SS after it + if "T" not in text: + return None + time_part = text.split("T", 1)[1] + # time_part must start with HH:MM:SS (8 chars minimum) + if len(time_part) < 8 or time_part[2] != ":" or time_part[5] != ":": + return None + try: + return datetime.fromisoformat(text) + except (ValueError, TypeError): + return None + + +def _interpolate_timestamps(items: list[dict]) -> list[dict]: + """Fill missing ``created_at`` fields via linear interpolation. + + Rules (backward-compatible — returns *items* unchanged when no message + carries a ``created_at`` value): + + 1. If **no** message has ``created_at`` → return as-is (system time used + later by AgentScope's ``Msg`` constructor). + 2. Messages **before** the first timestamped message → inherit the first + timestamp. + 3. Messages **after** the last timestamped message → inherit the last + timestamp. + 4. Messages **between** two timestamped anchors → linearly interpolated. + """ + # Pass 1: normalize aliases and collect anchors + normalized: list[dict] = [] + anchors: list[tuple[int, datetime]] = [] # (index, parsed_dt) + + for i, raw_item in enumerate(items): + if not isinstance(raw_item, dict): + normalized.append(raw_item) + continue + item = _normalize_msg_timestamp(raw_item) + normalized.append(item) + ca = item.get("created_at") + if ca: + dt = _parse_iso_seconds(str(ca)) + if dt is not None: + anchors.append((i, dt)) + + # No anchors → fully backward-compatible, let Msg use system time + if not anchors: + return normalized + + # Pass 2: interpolate + result: list[dict] = [] + for i, item in enumerate(normalized): + if not isinstance(item, dict): + result.append(item) + continue + # Already has a valid parsed anchor — keep it + if any(idx == i for idx, _ in anchors): + result.append(item) + continue + + # Find the nearest preceding and following anchors + prev_anchor: tuple[int, datetime] | None = None + next_anchor: tuple[int, datetime] | None = None + for idx, dt in anchors: + if idx < i: + prev_anchor = (idx, dt) + for idx, dt in anchors: + if idx > i: + next_anchor = (idx, dt) + break + + # Determine interpolated time + if prev_anchor is None: + # Before the first anchor + interpolated_dt = anchors[0][1] + elif next_anchor is None: + # After the last anchor + interpolated_dt = anchors[-1][1] + else: + # Between two anchors — linear + prev_idx, prev_dt = prev_anchor + next_idx, next_dt = next_anchor + span = next_idx - prev_idx + ratio = (i - prev_idx) / span + delta_seconds = (next_dt - prev_dt).total_seconds() + interpolated_dt = prev_dt + timedelta(seconds=delta_seconds * ratio) + + item = {**item, "created_at": interpolated_dt.isoformat()} + result.append(item) + + return result + + +@R.register("beam_auto_memory_step") +class BeamAutoMemoryStep(AutoMemoryStep): + """AutoMemoryStep variant that interpolates timestamps and pins daily_write to the resolved day.""" + + def _build_messages(self, raw_messages: list) -> list[Msg]: + # Interpolate timestamps: if any message carries created_at, fill in + # the rest via linear interpolation so the whole session has coherent + # time ordering (see _interpolate_timestamps docstring for rules). + interpolated = _interpolate_timestamps( + [item if not isinstance(item, dict) else dict(item) for item in raw_messages], + ) + return [self._to_msg(item) for item in interpolated] + + def _reply_extra_kwargs(self, day: str) -> dict: + return {"tool_defaults": {"daily_write": {"date": day}}} diff --git a/reme/steps/benchmark/beam/auto_memory.yaml b/reme/steps/benchmark/beam/auto_memory.yaml new file mode 100644 index 00000000..a9c59c27 --- /dev/null +++ b/reme/steps/benchmark/beam/auto_memory.yaml @@ -0,0 +1,334 @@ +system_prompt: | + You are an automatic memory extraction system. Your task is to extract ALL valuable information from conversations and record them into a structured daily memory note. Ensure **complete coverage** — every piece of information worth preserving must be captured. + + ## Extraction Principles + + 1. **Complete Coverage**: Generate enough content to ensure ALL information in the dialogues is captured. Missing information is worse than redundancy. + 2. **Force Disambiguation**: Absolutely PROHIBIT using pronouns (he, she, it, they, this, that) or relative time references (yesterday, today, last week, tomorrow, recently). Always use full names, explicit dates, and specific references. + 3. **Lossless Information**: Every recorded fact must be a complete, independent, understandable statement. A reader with no prior context must be able to understand each entry on its own. + 4. **Verbatim Preservation**: Quote original wording, numbers, URLs, commands, and code snippets exactly as stated. + + ## What to Extract + + - **Persons**: All person names mentioned (full names when available) + - **Entities**: Companies, products, organizations, projects, repositories + - **Decisions & Events**: What happened, what was decided, and why + - **Facts & Preferences**: Persistent facts about the user — identity, habits, preferences, constraints + - **Procedures & Solutions**: Actionable steps, commands, configurations that can be directly reused + - **Current State**: Progress, blockers, next steps, open questions + - **Temporal Information**: Use absolute dates in ISO 8601 format (YYYY-MM-DD) when explicit time is mentioned + - **Locations**: Specific location names when mentioned + - **Topic Keywords**: Core keywords that identify the subject matter + + ## Body Format + + Use structured sections with headings. Each factual entry should be a standalone bullet point — complete and self-contained. Group entries by topic. Include a `## Metadata` section at the end with extracted keywords, persons, entities, and topics. + + ## Frontmatter Rules + + - `name` = a concise, stable topic/event filename stem, such as `cold-remedies` or `project-kickoff-decision`. Do not include today's date or the daily directory date; the outer daily path already records the date. For existing notes, update it when a better filename is clearly warranted. + - `description` = a thorough, lossless summary; vague descriptions like "notes" / "misc" are unacceptable. The description alone should convey all key information. + - **Never set `status`** — it is a field reserved for downstream processing. + + ## Tool Call Format — CRITICAL + + You have access to tools (e.g. `daily_write`, `read`, `edit`, `write`, `frontmatter_update`). You MUST invoke them as **structured tool calls** (function calls), NOT as plain text output. + + - **NEVER** write tool invocations as text strings such as `daily_write name=...`. Text output does NOT execute tools — no memory will be saved. + - **ALWAYS** use the tool call mechanism provided by the runtime. The system handles parameter passing automatically. + - If you write a tool name and its parameters as text, NO tool will be executed and NO memory will be saved. +system_prompt_zh: | + 你是自动记忆提取系统。你的任务是从对话中提取所有有价值的信息，并记录到结构化的日记记忆中。确保**完整覆盖**——每一条值得保留的信息都必须被捕获。 + + ## 提取原则 + + 1. **完整覆盖**：生成足够的内容以确保对话中的所有信息都被捕获。遗漏信息比冗余更糟。 + 2. **强制消歧**：绝对禁止使用代词（他、她、它、他们、这个、那个）或相对时间（昨天、今天、上周、明天、最近）。必须使用全名、明确日期和具体引用。 + 3. **无损信息**：每条记录的事实必须是完整的、独立的、可理解的陈述。没有上下文的读者必须能单独理解每条记录。 + 4. **原文保留**：逐字引用原始措辞、数字、URL、命令和代码片段。 + + ## 提取什么 + + - **人物**：提及的所有人名（尽量使用全名） + - **实体**：公司、产品、组织、项目、仓库 + - **决策与事件**：发生了什么、决定了什么、为什么 + - **事实与偏好**：关于用户的持久事实——身份、习惯、偏好、约束 + - **流程与方案**：可直接复用的操作步骤、命令、配置 + - **当前状态**：进度、卡点、下一步、未决问题 + - **时间信息**：当对话中明确提及时间时，使用 ISO 8601 绝对日期格式（YYYY-MM-DD） + - **地点**：提及的具体地点名称 + - **主题关键词**：标识主题的核心关键词 + + ## 正文格式 + + 使用带标题的结构化分节。每条事实记录应是独立的列表项——完整且自包含。按主题分组。在末尾添加 `## Metadata` 部分，列出提取的关键词、人物、实体和主题。 + + ## Frontmatter 规则 + + - `name` = 简洁、稳定的主题/事件文件名 stem，例如 `cold-remedies` 或 `project-kickoff-decision`。不要包含今天日期或日记目录日期；外层日记路径已经记录日期。对已有笔记，如果明显有更好的文件名，就更新它。 + - `description` = 详尽的无损总结；模糊的描述如 "notes" / "misc" 不可接受。仅凭 description 就应能传达全部核心信息。 + - **永远不要设置 `status`**——它是下游处理保留的字段。 + + ## 工具调用格式 — 关键 + + 你可以使用工具（如 `daily_write`、`read`、`edit`、`write`、`frontmatter_update`）。你**必须**以**结构化工具调用**（函数调用）的方式调用它们，而不是以纯文本输出。 + + - **绝不**将工具调用作为文本字符串输出，如 `daily_write name=...`。文本输出**不会**执行工具——不会保存任何记忆。 + - **始终**使用运行时提供的工具调用机制。系统会自动处理参数传递。 + - 如果你将工具名和参数作为文本输出，**不会有任何工具被执行**，也**不会**保存任何记忆。 + +user_message_create: | + Today: {today} + Extra hint: {note} + Session ID: {session_id} + + # Recent Conversation + + {history} + + # Your Task + + Extract all valuable information from the conversation above and record them into a new daily memory note. + + ## Step 1 — Skip Check + + Did the conversation produce substantive information worth long-term memory? Pure greetings or small talk → reply with a brief skip message and stop (do not call any tools). + + When truly ambiguous, default to writing — losing a memory is worse than writing one extra note. + + ## Step 2 — Extract & Write + + Analyze the conversation and extract: + - **keywords**: Core keywords (names, places, entities, topic words) + - **persons**: All person names mentioned + - **entities**: Companies, products, organizations, projects + - **topic**: The main topic(s) of the conversation + - **timestamp**: Absolute time in ISO 8601 format (if explicit time mentioned) + - **location**: Specific location name (if mentioned) + + Then create the note by **calling the `daily_write` tool** with these parameters: + - `name`: + - `description`: + - `session_id`: {session_id} + - `date`: {today} + - `content`: + + ⚠️ CRITICAL: You MUST invoke `daily_write` as a **tool call** (function call). Do NOT write `daily_write name=...` as plain text — that will NOT execute the tool and NO memory will be saved. + + Body requirements: + - Each factual entry must be a complete, independent, self-contained statement (lossless restatement). + - PROHIBIT pronouns (he/she/it/they/this/that) — use full names and explicit references. + - PROHIBIT relative time (yesterday/today/last week) — use absolute dates (YYYY-MM-DD). + - Group entries by topic with headings. + - End with a `## Metadata` section containing: keywords, persons, entities, topic. + + Filename and description rules: + - `name`: concise, stable topic/event filename stem in kebab-case. No date, no slash/backslash, no `< > : " | ? *`. + - `description`: a thorough, lossless summary — specific enough that the description alone conveys all key information. + + ## Step 3 — Summary + + State in one sentence what you did (which file was created). This is your final text output. + + ## Boundaries + + - Create at most 3 notes for this session. If the conversation covers multiple distinct topics, split them into separate notes (one topic per note). If only one topic is discussed, create just one note. +user_message_create_zh: | + 今天：{today} + 额外提示：{note} + Session ID：{session_id} + + # 最近的对话 + + {history} + + # 你的任务 + + 从上述对话中提取所有有价值的信息，并记录到一条新的日记记忆中。 + + ## 步骤 1 — 跳过检查 + + 对话是否产生了值得长期记忆的实质性信息？纯粹的寒暄或闲聊 → 回复一条简短的跳过消息并停止（不调用任何工具）。 + + 当真正模棱两可时，默认写入——丢失记忆比多写一条笔记更糟。 + + ## 步骤 2 — 提取并写入 + + 分析对话并提取： + - **keywords**：核心关键词（人名、地点、实体、主题词） + - **persons**：提及的所有人名 + - **entities**：公司、产品、组织、项目 + - **topic**：对话的主要主题 + - **timestamp**：ISO 8601 格式的绝对时间（如果对话中明确提及时间） + - **location**：具体地点名称（如果提及） + + 然后**调用 `daily_write` 工具**，使用以下参数一次性创建笔记： + - `name`：<简洁、稳定的主题/事件文件名 stem，采用 kebab-case> + - `description`：<详尽的无损总结> + - `session_id`：{session_id} + - `date`：{today} + - `content`：<完整正文> + + ⚠️ 关键：你**必须**以**工具调用**（函数调用）方式调用 `daily_write`。不要将 `daily_write name=...` 作为纯文本输出——那样**不会**执行工具，也**不会**保存任何记忆。 + + 正文要求： + - 每条事实记录必须是完整的、独立的、自包含的陈述（无损复述）。 + - 禁止使用代词（他/她/它/他们/这个/那个）——必须使用全名和明确引用。 + - 禁止使用相对时间（昨天/今天/上周）——必须使用绝对日期（YYYY-MM-DD）。 + - 按主题分组并使用标题。 + - 末尾添加 `## Metadata` 部分，包含：keywords、persons、entities、topic。 + + 文件名和描述规则： + - `name`：简洁、稳定的主题/事件文件名 stem，采用 kebab-case。不含日期、不含 slash/反斜杠、不含 `< > : " | ? *`。 + - `description`：详尽的无损总结——具体到仅凭 description 就能传达全部核心信息。 + + ## 步骤 3 — 总结 + + 用一句话说明你做了什么（创建了哪个文件）。这是你最后一次文本输出。 + + ## 边界 + + - 当前 session 最多创建 3 条笔记。如果对话涵盖多个不同主题，将其拆分为独立笔记（每个主题一条）。如果只讨论了一个主题，则只创建一条笔记。 + +user_message_update: | + Today: {today} + Extra hint: {note} + Target path: {note_path} + + # Recent Conversation + + {history} + + # Your Task + + Extract all valuable information from the conversation above and merge them into the existing daily note at the target path. + + ## Step 1 — Skip Check + + Did the conversation produce substantive information worth long-term memory? Pure greetings or small talk → reply with a brief skip message and stop (do not call any tools). + + When truly ambiguous, default to writing — losing a memory is worse than writing one extra note. + + ## Step 2 — Read Existing Content + + **Call the `read` tool** with parameter `path` set to `{note_path}` to inspect the current note content. You MUST invoke it as a tool call, not as text. + - If the body is empty (only frontmatter, no actual content) → treat as new, jump to **Step 3b**. + - If there is body content → go to **Step 3a** to merge. + + ## Step 3a — Merge Update + + The note already has content. Extract new information from the conversation and merge into it. + + Extraction rules (apply to ALL new entries): + - Each new fact must be a complete, independent, self-contained statement (lossless restatement). + - PROHIBIT pronouns (he/she/it/they/this/that) — use full names and explicit references. + - PROHIBIT relative time (yesterday/today/last week) — use absolute dates (YYYY-MM-DD). + + Merge rules: + - **Timeline / history entries**: append only, never delete existing entries. + - **Current-state entries** (progress, blockers, next steps, open questions): rewrite the entire section to reflect the latest snapshot. + - **Everything else**: merge and deduplicate — keep all old facts, add new facts, remove exact duplicates. + - **Metadata section**: merge keywords, persons, entities, topics — union of old and new values. + + Execution: + 1. **Call the `edit` tool** for each section that needs updating, with parameters: `path`="{note_path}", `old`=, `new`=. You may call `edit` multiple times. Each call MUST be a tool call, not text. + 2. After body changes, **call the `frontmatter_update` tool** to refresh frontmatter, with parameters: `path`="{note_path}", `metadata`={{"name": "", "description": ""}}. + - Keep the existing `name` only when it is already the best concise topic/event filename stem. The system will rename the file after your final response. + - Do not add today's date or the daily directory date to `name`. + 3. If `edit` fails repeatedly (e.g., cannot find the original text due to formatting mismatch), fall back to **calling the `write` tool** with parameters: `path`="{note_path}", `name`=, `description`=, `content`= for a complete rewrite. + + ## Step 3b — Full Write (Empty File Fallback) + + The file exists but its body is empty. Extract and write the full content by **calling the `write` tool** with parameters: `path`="{note_path}", `name`=, `description`=, `content`=. + + Body requirements: + - Each factual entry must be a complete, independent, self-contained statement. + - PROHIBIT pronouns — use full names and explicit references. + - PROHIBIT relative time — use absolute dates (YYYY-MM-DD). + - Group entries by topic with headings. + - End with a `## Metadata` section containing: keywords, persons, entities, topic. + + Filename and description rules: + - `name`: concise, stable topic/event filename stem in kebab-case. No date. + - `description`: a thorough, lossless summary — the description alone should convey all key information. + + ## Step 4 — Summary + + State in one sentence what you did (what content was updated). This is your final text output. + + ## Boundaries + + - Only operate on one target path: `{note_path}`. Do not touch other notes. + - `write` unconditionally overwrites body and frontmatter — use with caution. + - Filename changes are expressed by updating frontmatter `name`. +user_message_update_zh: | + 今天：{today} + 额外提示：{note} + 目标路径：{note_path} + + # 最近的对话 + + {history} + + # 你的任务 + + 从上述对话中提取所有有价值的信息，并合并到目标路径的已有日记中。 + + ## 步骤 1 — 跳过检查 + + 对话是否产生了值得长期记忆的实质性信息？纯粹的寒暄或闲聊 → 回复一条简短的跳过消息并停止（不调用任何工具）。 + + 当真正模棱两可时，默认写入——丢失记忆比多写一条笔记更糟。 + + ## 步骤 2 — 读取现有内容 + + **调用 `read` 工具**，参数 `path` 设为 `{note_path}`，查看当前笔记内容。必须以工具调用方式调用，不要作为文本输出。 + - 如果正文为空（只有 frontmatter 无实际内容）→ 按新建处理，跳到 **步骤 3b**。 + - 如果有正文内容 → 转到 **步骤 3a** 进行合并。 + + ## 步骤 3a — 合并更新 + + 笔记已有内容。从对话中提取新信息并合并进去。 + + 提取规则（适用于所有新条目）： + - 每条新事实必须是完整的、独立的、自包含的陈述（无损复述）。 + - 禁止使用代词（他/她/它/他们/这个/那个）——必须使用全名和明确引用。 + - 禁止使用相对时间（昨天/今天/上周）——必须使用绝对日期（YYYY-MM-DD）。 + + 合并规则： + - **时间线 / 历史条目**：仅追加，永远不删除已有条目。 + - **当下状态类条目**（进度、卡点、下一步、未决问题）：整段重写，反映最新快照。 + - **其余内容**：合并去重——保留全部旧事实，添加新事实，去除完全重复项。 + - **Metadata 部分**：合并 keywords、persons、entities、topics——取旧值和新值的并集。 + + 执行： + 1. **调用 `edit` 工具**更新需要修改的部分，参数为：`path`="{note_path}"、`old`=<原文片段>、`new`=<替换片段>。可以多次调用 `edit`。每次调用必须是工具调用，不是文本。 + 2. 正文变更后，**调用 `frontmatter_update` 工具**刷新 frontmatter，参数为：`path`="{note_path}"、`metadata`={{"name": "<更新后的文件名 stem>", "description": "<更新后的总结>"}}。 + - 只有当前 `name` 已经是最合适的简洁主题/事件文件名 stem 时才保留。系统会在你最终回复后负责重命名文件。 + - 不要在 `name` 中加入今天日期或日记目录日期。 + 3. 如果 `edit` 多次失败（如因格式不匹配找不到原文），退回**调用 `write` 工具**全量重写，参数为：`path`="{note_path}"、`name`=、`description`=、`content`=<完整正文>。 + + ## 步骤 3b — 全量写入（空文件 fallback） + + 文件存在但正文为空。提取并**调用 `write` 工具**一次性写入完整内容，参数为：`path`="{note_path}"、`name`=、`description`=、`content`=<正文>。 + + 正文要求： + - 每条事实记录必须是完整的、独立的、自包含的陈述。 + - 禁止使用代词——必须使用全名和明确引用。 + - 禁止使用相对时间——必须使用绝对日期（YYYY-MM-DD）。 + - 按主题分组并使用标题。 + - 末尾添加 `## Metadata` 部分，包含：keywords、persons、entities、topic。 + + 文件名和描述规则： + - `name`：简洁、稳定的主题/事件文件名 stem，采用 kebab-case。不含日期。 + - `description`：详尽的无损总结——仅凭 description 就应能传达全部核心信息。 + + ## 步骤 4 — 总结 + + 用一句话说明你做了什么（更新了哪些内容）。这是你最后一次文本输出。 + + ## 边界 + + - 只针对一个目标路径：`{note_path}`。不要碰其他笔记。 + - `write` 会无条件覆盖正文和 frontmatter，请谨慎使用。 + - 文件名变化通过更新 frontmatter `name` 表达。 diff --git a/reme/steps/benchmark/beam/llm_judge.py b/reme/steps/benchmark/beam/llm_judge.py new file mode 100644 index 00000000..bf51f345 --- /dev/null +++ b/reme/steps/benchmark/beam/llm_judge.py @@ -0,0 +1,366 @@ +"""BEAM rubric-based LLM-as-Judge step. + +For each rubric item, calls the judge LLM with the unified judge prompt +and collects scores (0.0 / 0.5 / 1.0). The final ``llm_judge_score`` is +the average across all rubric items. + +For ``event_ordering`` questions, additionally computes: + - LLM-based event alignment (matching system events to reference events) + - precision / recall / f1 (set-intersection after alignment) + - Kendall's tau (ordering correlation, pure numpy implementation) + - final_score = tau_norm * f1 + +A ``semantic`` alignment path is also available which uses ReMe's +configured ``as_embedding`` model (replacing BEAM's sentence_transformers). + +This replicates the evaluation logic from +``benchmark/datasets/BEAM/src/evaluation/compute_metrics.py``. +""" + +import json +import re +from typing import List, Tuple + +import numpy as np +from json_repair import repair_json + +from ...base_step import BaseStep, Ref +from ....components import R +from ....components.as_embedding import BaseAsEmbedding +from ....enumeration import ComponentEnum + + +# --------------------------------------------------------------------------- +# JSON parsing helper (replicates BEAM's parse_json_response) +# --------------------------------------------------------------------------- +def _parse_json_response(response: str) -> dict: + response = response.strip() + + if response.startswith("```"): + match = re.search( + r"```(?:json)?\s*(\[.*\]|\{.*\})\s*```", + response, + re.DOTALL, + ) + if match: + response = match.group(1).strip() + + try: + return json.loads(response) + except json.JSONDecodeError: + pass + + match = re.search(r"(\{.*?\}|\[.*?\])", response, re.DOTALL) + if match: + json_part = match.group(1) + try: + return json.loads(json_part) + except Exception as e: + raise ValueError(f"Found possible JSON but failed to parse it: {e}") from e + + raise ValueError("No valid JSON found in response.") + + +# --------------------------------------------------------------------------- +# Event-ordering helpers (replicate BEAM's compute_metrics.py) +# --------------------------------------------------------------------------- +async def _llm_equivalence(agent_wrapper, reference: str, system: str) -> bool: + """Binary classifier: do the two snippets describe the SAME event/fact? + + Replicates BEAM's ``llm_equivalence`` using ``agent_wrapper.reply()``. + """ + system_prompt = ( + "You are a binary classifier.\n" + "If the TWO snippets describe the SAME event/fact, reply **YES**\n" + "Otherwise reply **NO**. No extra words.\n" + "DO NOT provide any explanation." + ) + user_prompt = f"First snippet: {reference}\n\nSecond snippet: {system}" + + result = await agent_wrapper.reply(user_prompt, system_prompt=system_prompt) + raw = (result.get("result") or "").strip().lower() + return "yes" in raw + + +async def _align_with_llm( + agent_wrapper, + reference: List[str], + system: List[str], +) -> Tuple[List[str], List[str]]: + """Align system events to reference events via LLM equivalence. + + Replicates BEAM's ``align_with_llm``: for each system event, find the + first unmatched reference event that is LLM-equivalent. If found, + replace the system event with the reference text (canonicalisation). + Ensures 1-to-1 mapping. + """ + used = set() + system_out = [] + + for s in system: + matched_index = None + for index, r in enumerate(reference): + if index in used: + continue + if await _llm_equivalence(agent_wrapper, reference=r, system=s): + matched_index = index + break + + if matched_index is not None: + system_out.append(reference[matched_index]) + used.add(matched_index) + else: + system_out.append(s) + + return reference, system_out + + +async def _semantic_align( + embedding_fn, + reference: List[str], + system: List[str], + thr: float = 0.65, +) -> Tuple[List[str], List[str]]: + """Align system events to reference events via embedding cosine similarity. + + Replaces BEAM's ``semantic_align`` (which used sentence_transformers) + with ReMe's configured ``as_embedding`` model. + """ + if not reference or not system: + return reference, system + + ref_embeddings = np.array(await embedding_fn(reference)) + sys_embeddings = np.array(await embedding_fn(system)) + + # Normalise + ref_norms = ref_embeddings / (np.linalg.norm(ref_embeddings, axis=1, keepdims=True) + 1e-12) + sys_norms = sys_embeddings / (np.linalg.norm(sys_embeddings, axis=1, keepdims=True) + 1e-12) + + used_reference = set() + system_canon = [] + + for i, s_txt in enumerate(system): + sims = sys_norms[i] @ ref_norms.T # cosine similarity + best = int(np.argmax(sims)) + if sims[best] >= thr and best not in used_reference: + system_canon.append(reference[best]) + used_reference.add(best) + else: + system_canon.append(s_txt) + + return reference, system_canon + + +def _kendall_tau_b(x: list, y: list) -> float: + """Compute Kendall's tau-b rank correlation using only numpy. + + Replicates ``scipy.stats.kendalltau(x, y, variant='b')`` for the + rank-based inputs used in event ordering scoring. + """ + x_arr = np.asarray(x, dtype=float) + y_arr = np.asarray(y, dtype=float) + n = len(x_arr) + if n < 2: + return 0.0 + + concordant = 0 + discordant = 0 + x_ties = 0 + y_ties = 0 + + for i in range(n - 1): + for j in range(i + 1, n): + dx = x_arr[j] - x_arr[i] + dy = y_arr[j] - y_arr[i] + if dx == 0 and dy == 0: + x_ties += 1 + y_ties += 1 + elif dx == 0: + x_ties += 1 + elif dy == 0: + y_ties += 1 + elif (dx > 0) == (dy > 0): + concordant += 1 + else: + discordant += 1 + + n0 = n * (n - 1) / 2 + denom = np.sqrt((n0 - x_ties) * (n0 - y_ties)) + if denom == 0: + return 0.0 + return (concordant - discordant) / denom + + +def _event_ordering_score( + reference_canon: List[str], + system_canon: List[str], +) -> dict: + """Compute precision/recall/f1 + Kendall's tau after alignment. + + Replicates BEAM's ``event_ordering_score`` (the scoring part, after + alignment is done). + """ + tp = len(set(reference_canon) & set(system_canon)) + fp = len([x for x in system_canon if x not in reference_canon]) + fn = len([x for x in reference_canon if x not in system_canon]) + + precision = tp / (tp + fp) if tp + fp else 0 + recall = tp / (tp + fn) if tp + fn else 0 + f1 = 2 * precision * recall / (precision + recall) if (precision + recall) else 0 + + union = list(dict.fromkeys(reference_canon + system_canon)) + tie_rank = len(union) + 1 + + def to_rank(seq): + r = {item: i + 1 for i, item in enumerate(seq)} + return [r.get(u, tie_rank) for u in union] + + tau_b = _kendall_tau_b( + to_rank(reference_canon), + to_rank(system_canon), + ) + tau_b_norm = (tau_b + 1) / 2 if tau_b is not None else 0 + + final_score = tau_b_norm * f1 + return { + "precision": precision, + "recall": recall, + "f1": f1, + "tau_norm": tau_b_norm, + "final_score": final_score, + } + + +@R.register("beam_rubric_judge_step") +class BeamRubricJudgeStep(BaseStep): + """Judge an LLM response against a list of rubric criteria. + + Inputs (from RuntimeContext): + llm_response (str, required): The model's response to evaluate. + rubric (list[str], required): Rubric criteria to check. + probing_question (str, optional): The original probing question. + question_type (str, optional): Question type (e.g. "event_ordering"). + + Output (written to context.response): + answer = str(llm_judge_score) + metadata["llm_judge_score"] = float + metadata["llm_judge_responses"] = list[dict] + metadata["event_ordering"] = dict (only for event_ordering type) + """ + + as_embedding: BaseAsEmbedding = Ref( + BaseAsEmbedding, + ComponentEnum.AS_EMBEDDING, + optional=True, + ) + + async def execute(self): + assert self.context is not None + llm_response: str = self.context.get("llm_response", "") + rubric: list[str] = self.context.get("rubric", []) + probing_question: str = self.context.get("probing_question", "") + question_type: str = self.context.get("question_type", "") + + if not llm_response: + raise ValueError("beam_rubric_judge_step requires non-empty llm_response") + if not rubric: + raise ValueError("beam_rubric_judge_step requires non-empty rubric") + if self.agent_wrapper is None: + raise RuntimeError("beam_rubric_judge_step requires agent_wrapper") + + # ----- Standard rubric-based LLM-as-Judge (all question types) ----- + judge_template = self.get_prompt("judge_prompt") + + llm_judge_responses: list[dict] = [] + total_score = 0.0 + + for item in rubric: + prompt = judge_template.replace("", item).replace("", llm_response) + + result = await self.agent_wrapper.reply(prompt) + raw = (result.get("result") or "").strip() + + try: + parsed = _parse_json_response(raw) + except Exception: + try: + parsed = json.loads(repair_json(raw)) + except Exception: + parsed = {"score": 0.0, "reason": f"Failed to parse: {raw[:200]}"} + + score = float(parsed.get("score", 0)) + + # Abstention: binary classification — 1.0 stays 1, <1.0 becomes 0 + if question_type == "abstention": + score = 1.0 if score >= 1.0 else 0.0 + + total_score += score + llm_judge_responses.append(parsed) + + llm_judge_score = total_score / len(rubric) if rubric else 0.0 + + self.logger.info(f"[{self.name}] judge score: {llm_judge_score:.3f}") + + self.context.response.success = True + self.context.response.answer = str(llm_judge_score) + self.context.response.metadata.update( + { + "llm_judge_score": llm_judge_score, + "llm_judge_responses": llm_judge_responses, + "rubric": rubric, + "llm_response": llm_response, + "probing_question": probing_question, + "question_type": question_type, + }, + ) + + # ----- event_ordering extra metrics ----- + # Replicates BEAM's evaluate_event_ordering: system_list = llm_response.split("\n") + # Note: BEAM calls extract_facts first but immediately overwrites with split("\n"). + if question_type == "event_ordering": + eo_metrics = await self._compute_event_ordering( + rubric=rubric, + llm_response=llm_response, + ) + self.context.response.metadata["event_ordering"] = eo_metrics + self.logger.info(f"[{self.name}] event_ordering: {eo_metrics}") + + return self.context.response + + async def _compute_event_ordering( + self, + rubric: list[str], + llm_response: str, + ) -> dict: + """Compute event_ordering extra metrics. + + Uses ``align_type="llm"`` to match BEAM's original code. + Also supports ``align_type="semantic"`` via ReMe's embedding model + (replacing sentence_transformers). + """ + # BEAM: system_list = llm_response.split("\n") + system_list = [line for line in llm_response.split("\n") if line.strip()] + + # Use LLM alignment (matching BEAM's align_type="llm") + reference_canon, system_canon = await _align_with_llm( + agent_wrapper=self.agent_wrapper, + reference=rubric, + system=system_list, + ) + + eo_score = _event_ordering_score(reference_canon, system_canon) + + # Also compute semantic alignment if embedding is available + if self.as_embedding is not None: + try: + ref_canon_sem, sys_canon_sem = await _semantic_align( + embedding_fn=self.as_embedding, + reference=rubric, + system=system_list, + ) + eo_score_sem = _event_ordering_score(ref_canon_sem, sys_canon_sem) + eo_score["semantic_alignment"] = eo_score_sem + except Exception as e: + self.logger.warning(f"[{self.name}] semantic_align failed: {e}") + + return eo_score diff --git a/reme/steps/benchmark/beam/llm_judge.yaml b/reme/steps/benchmark/beam/llm_judge.yaml new file mode 100644 index 00000000..cfef535a --- /dev/null +++ b/reme/steps/benchmark/beam/llm_judge.yaml @@ -0,0 +1,70 @@ +judge_prompt: | + You are an expert evaluator tasked with judging whether the LLM's response demonstrates compliance with the specified RUBRIC CRITERION. + + ## EVALUATION INPUTS + - QUESTION (what the user asked): + - RUBRIC CRITERION (what to check): + - RESPONSE TO EVALUATE: + + ## EVALUATION RUBRIC: + The rubric defines a specific requirement, constraint, or expected behavior that the LLM response should demonstrate. + + **IMPORTANT**: Pay careful attention to whether the rubric specifies: + - **Positive requirements** (things the response SHOULD include/do) + - **Negative constraints** (things the response SHOULD NOT include/do, often indicated by "no", "not", "avoid", "absent") + + ## RESPONSIVENESS REQUIREMENT (anchored to the QUESTION) + A compliant response must be **on-topic with respect to the QUESTION** and attempt to answer it. + - If the response does not address the QUESTION, score **0.0** and stop. + - For negative constraints, both must hold: (a) the response is responsive to the QUESTION, and (b) the prohibited element is absent. + + ## SEMANTIC TOLERANCE RULES: + Judge by meaning, not exact wording. + - Accept **paraphrases** and **synonyms** that preserve intent. + - **Case/punctuation/whitespace** differences must be ignored. + - **Numbers/currencies/dates** may appear in equivalent forms (e.g., "$68,000", "68k", "68,000 USD", or "sixty-eight thousand dollars"). Treat them as equal when numerically equivalent. + - If the rubric expects a number or duration, prefer **normalized comparison** (extract and compare values) over string matching. + + ## STYLE NEUTRALITY (prevents style contamination): + Ignore tone, politeness, length, and flourish unless the rubric explicitly requires a format/structure (e.g., "itemized list", "no citations", "one sentence"). + - Do **not** penalize hedging, voice, or verbosity if content satisfies the rubric. + - Only evaluate format when the rubric **explicitly** mandates it. + + ## SCORING SCALE: + - **1.0 (Complete Compliance)**: Fully complies with the rubric criterion. + - Positive: required element present, accurate, properly executed (allowing semantic equivalents). + - Negative: prohibited element **absent** AND response is **responsive**. + + - **0.5 (Partial Compliance)**: Partially complies. + - Positive: element present but minor inaccuracies/incomplete execution. + - Negative: generally responsive and mostly avoids the prohibited element but with minor/edge violations. + + - **0.0 (No Compliance)**: Fails to comply. + - Positive: required element missing or incorrect. + - Negative: prohibited element present **or** response is non-responsive/evasive even if the element is absent. + + ## EVALUATION INSTRUCTIONS: + 1. **Understand the Requirement**: Determine if the rubric is asking for something to be present (positive) or absent (negative/constraint). + + 2. **Parse Compound Statements**: If the rubric contains multiple elements connected by "and" or commas, evaluate whether: + - **All elements** must be present for full compliance (1.0) + - **Some elements** present indicates partial compliance (0.5) + - **No elements** present indicates no compliance (0.0) + + 3. **Check Compliance**: + - For positive requirements: Look for the presence and quality of the required element + - For negative constraints: Look for the absence of the prohibited element + + 4. **Assign Score**: Based on compliance with the specific rubric criterion according to the scoring scale above. + + 5. **Provide Reasoning**: Explain whether the rubric criterion was satisfied and justify the score. + + ## OUTPUT FORMAT: + Return your evaluation in JSON format with two fields: + + { + "score": [your score: 1.0, 0.5, or 0.0], + "reason": "[detailed explanation of whether the rubric criterion was satisfied and why this justified the assigned score]" + } + + NOTE: ONLY output the json object, without any explanation before or after that diff --git a/reme/steps/benchmark/lme/__init__.py b/reme/steps/benchmark/lme/__init__.py index aba19f47..29005cc7 100644 --- a/reme/steps/benchmark/lme/__init__.py +++ b/reme/steps/benchmark/lme/__init__.py @@ -1,21 +1,11 @@ """LongMemEval benchmark steps.""" from .agentic_answer import LmeAgenticAnswerStep +from .llm_judge import LmeAnswerJudgeStep from .auto_memory import LmeAutoMemoryStep -from .context_answer import ContextAnswerStep -from .extract_session import LmeExtractSessionStep -from .final_answer_review import FinalAnswerReviewStep -from .golden_check import GoldenCheckStep -from .lme_llm_judge import LmeLlmJudgeStep -from .session_review import SessionReviewStep __all__ = [ - "ContextAnswerStep", - "FinalAnswerReviewStep", - "GoldenCheckStep", "LmeAgenticAnswerStep", + "LmeAnswerJudgeStep", "LmeAutoMemoryStep", - "LmeExtractSessionStep", - "LmeLlmJudgeStep", - "SessionReviewStep", ] diff --git a/reme/steps/benchmark/lme/agentic_answer.py b/reme/steps/benchmark/lme/agentic_answer.py index fd90952d..add3a676 100644 --- a/reme/steps/benchmark/lme/agentic_answer.py +++ b/reme/steps/benchmark/lme/agentic_answer.py @@ -1,87 +1,16 @@ -"""agentic_answer — answer the LongMemEval question from the indexed memory. +"""LongMemEval agentic answer step – ReAct agent that answers questions using the search tool.""" -Job #4 of the pipeline. Reads ``query.json`` and hands the question to an agent -equipped with ``vector_search`` / ``bm25_search`` / ``python_execute`` / -``extract_session_by_id``. The agent searches the daily-note index, pivots to -raw sessions by ``session_id`` when a hit is promising, and keeps trying until it -can answer or has searched too many times. The final answer is written to -``mem_answer.json`` in the workspace. -""" - -import json - -from ...base_step import BaseStep from ....components import R +from ..base import BaseAgenticAnswerStep @R.register("lme_agentic_answer_step") -class LmeAgenticAnswerStep(BaseStep): - """Drive the tool-using agent that answers from indexed memory.""" +class LmeAgenticAnswerStep(BaseAgenticAnswerStep): + """Answer a LongMemEval query via ReAct agent with access to the search tool. - _OUTPUT_FILE = "mem_answer.json" + The agent uses the ``agent_wrapper`` component in ReAct mode, calling the + ``search`` job tool to retrieve relevant memory chunks before generating + a final answer. + """ - def _load_query(self) -> dict: - path = self.workspace_path / "query.json" - with path.open(encoding="utf-8") as f: - data = json.load(f) - if not isinstance(data, dict): - raise ValueError("query.json is not a JSON object") - return data - - async def execute(self): - assert self.context is not None - if self.agent_wrapper is None: - raise ValueError("lme_agentic_answer_step requires agent_wrapper") - - query = self._load_query() - question = str(query.get("question", "") or "").strip() - question_date = str(query.get("question_date", "") or "").strip() - question_id = str(query.get("question_id", "") or "").strip() - if not question: - raise ValueError("query.json requires a non-empty 'question'") - - user_prompt = self.prompt_format( - "user_message", - question=question, - question_date=question_date or "(unknown)", - ) - # A stable tool_context_id makes vector/bm25 dedup across this answer run, - # so repeated searches surface genuinely new chunks each time. - result = await self.agent_wrapper.reply( - user_prompt, - system_prompt=self.get_prompt("system_prompt"), - tool_context_id=question_id or question, - ) - answer = (result.get("result") or "").strip() - # session_id names the trajectory file mem_session/agentscope/.jsonl, - # so downstream tooling can locate this run's full tool-call trail. - session_id = str(result.get("session_id") or "") - - out_path = self.workspace_path / self._OUTPUT_FILE - out_path.write_text( - json.dumps( - { - "question_id": question_id, - "question": question, - "answer": answer, - "session_id": session_id, - }, - ensure_ascii=False, - indent=2, - ), - encoding="utf-8", - ) - - self.logger.info(f"[{self.name}] answer for {question_id or question!r}: {answer!r}") - self.context.response.success = True - self.context.response.answer = answer - self.context.response.metadata.update( - { - "question_id": question_id, - "question": question, - "answer": answer, - "session_id": session_id, - "path": self._OUTPUT_FILE, - }, - ) - return self.context.response + TOOL_CONTEXT_PREFIX = "lme_agentic_answer" diff --git a/reme/steps/benchmark/lme/agentic_answer.yaml b/reme/steps/benchmark/lme/agentic_answer.yaml index 36ccbf82..f856f0fc 100644 --- a/reme/steps/benchmark/lme/agentic_answer.yaml +++ b/reme/steps/benchmark/lme/agentic_answer.yaml @@ -1,40 +1,18 @@ system_prompt: | - You answer a user's question using ONLY their long-term memory, retrieved through tools. The - memory is a set of daily notes, each distilled from one past chat session and tagged in its - header with a `session_id` pointing back to the raw session. + You are a memory retrieval assistant. You MUST use the search tool to find information before answering. + - Your total time of tool calls should be at most 9 times - Available tools: - - `vector_search(query)`: dense semantic search over the memory notes. - - `bm25_search(query)`: keyword search over the memory notes. - - `extract_session_by_id(session_id)`: go back to the ORIGINAL raw session behind a note and - pull out its full content relevant to the question. Use the `session_id` shown in a search - result's header. - - `python_execute(code)`: run Python for any counting, date math, or reasoning over what you found. + ## Search Strategy + - You can call 'search' tool to search multiple times (at least once) with different queries to gather comprehensive information. + - Do not stop after the first relevant one unless you are confident that the information is sufficient. + ## Draft Tool + - Use 'add_draft' to save key findings during search, and 'read_all_draft' to review all saved notes before answering. + ## Answer Rules + - Answer based ONLY on retrieved context. + - If the memories contain contradictory information, integrate the context, query, and most importantly the timestamp of each memory to comprehensively determine the final answer. + - Output ONLY the direct factual answer — no reasoning, no search process, no elaboration. + - Do NOT invent or assume information that isn't in the memories. + - If information is not founded or not sufficient after multiple searches, reply: 'Information not found.' - Strategy: - 1. Start by searching with the user's ORIGINAL question wording — call BOTH `vector_search` and - `bm25_search` with it. - 2. If the results already fully support an answer, answer. - 3. If a result looks relevant but the distilled note is not enough (missing a number, exact date, - or wording), take its `session_id` and call `extract_session_by_id` to read the raw session. - 4. Do NOT give up early. Keep trying: reformulate with new keywords, aliases, entities, dates, and - short phrases from the question or from earlier results; search again; and call - `extract_session_by_id` on any additional relevant sessions. - 5. Search results are deduplicated within this run, so a search returning nothing new means those - chunks were already seen — change your wording rather than repeating it. - 6. Only after you have made MORE THAN 10 search attempts and still cannot find support, answer - exactly: not provided - - Answer rules: - - Answer strictly from retrieved memory; never invent facts. - - Be direct and specific; include the exact value/date the question asks for. - - Your final message is the answer itself (no tool calls, no preamble). - -user_message: | - Question date: {question_date} - Question: {question} - - Find the answer in the user's long-term memory using the tools, following the strategy above. - Remember: search with the original question first (both vector and bm25), pivot to - `extract_session_by_id` for promising sessions, keep trying with new wording, and only answer - "not provided" after more than 10 search attempts have failed. +temporal_hint: | + Current time context: {query_time} diff --git a/reme/steps/benchmark/lme/auto_memory.py b/reme/steps/benchmark/lme/auto_memory.py index fcd33df6..433a0d58 100644 --- a/reme/steps/benchmark/lme/auto_memory.py +++ b/reme/steps/benchmark/lme/auto_memory.py @@ -1,428 +1,132 @@ -"""lme_auto_memory — turn every LongMemEval session into a search-friendly note. +"""auto_memory (lme) — AutoMemoryStep with timestamp interpolation and daily_write date default.""" -For a workspace such as ``datasets/longmemeval/1`` this step walks each raw -session under ``resource_dir`` (files named ``_(...)_ + - `description`: + - `session_id`: {session_id} + - `date`: {today} + - `content`: + + ⚠️ CRITICAL: You MUST invoke `daily_write` as a **tool call** (function call). Do NOT write `daily_write name=...` as plain text — that will NOT execute the tool and NO memory will be saved. + + Body requirements: + - Each factual entry must be a complete, independent, self-contained statement (lossless restatement). + - PROHIBIT pronouns (he/she/it/they/this/that) — use full names and explicit references. + - PROHIBIT relative time (yesterday/today/last week) — use absolute dates (YYYY-MM-DD). + - Group entries by topic with headings. + - End with a `## Metadata` section containing: keywords, persons, entities, topic. + + Filename and description rules: + - `name`: concise, stable topic/event filename stem in kebab-case. No date, no slash/backslash, no `< > : " | ? *`. + - `description`: a thorough, lossless summary — specific enough that the description alone conveys all key information. + + ## Step 3 — Summary + + State in one sentence what you did (which file was created). This is your final text output. + + ## Boundaries + + - Create at most 3 notes for this session. If the conversation covers multiple distinct topics, split them into separate notes (one topic per note). If only one topic is discussed, create just one note. +user_message_create_zh: | + 今天：{today} + 额外提示：{note} + Session ID：{session_id} + + # 最近的对话 + + {history} + + # 你的任务 + + 从上述对话中提取所有有价值的信息，并记录到一条新的日记记忆中。 + + ## 步骤 1 — 跳过检查 + + 对话是否产生了值得长期记忆的实质性信息？纯粹的寒暄或闲聊 → 回复一条简短的跳过消息并停止（不调用任何工具）。 + + 当真正模棱两可时，默认写入——丢失记忆比多写一条笔记更糟。 + + ## 步骤 2 — 提取并写入 + + 分析对话并提取： + - **keywords**：核心关键词（人名、地点、实体、主题词） + - **persons**：提及的所有人名 + - **entities**：公司、产品、组织、项目 + - **topic**：对话的主要主题 + - **timestamp**：ISO 8601 格式的绝对时间（如果对话中明确提及时间） + - **location**：具体地点名称（如果提及） + + 然后**调用 `daily_write` 工具**，使用以下参数一次性创建笔记： + - `name`：<简洁、稳定的主题/事件文件名 stem，采用 kebab-case> + - `description`：<详尽的无损总结> + - `session_id`：{session_id} + - `date`：{today} + - `content`：<完整正文> + + ⚠️ 关键：你**必须**以**工具调用**（函数调用）方式调用 `daily_write`。不要将 `daily_write name=...` 作为纯文本输出——那样**不会**执行工具，也**不会**保存任何记忆。 + + 正文要求： + - 每条事实记录必须是完整的、独立的、自包含的陈述（无损复述）。 + - 禁止使用代词（他/她/它/他们/这个/那个）——必须使用全名和明确引用。 + - 禁止使用相对时间（昨天/今天/上周）——必须使用绝对日期（YYYY-MM-DD）。 + - 按主题分组并使用标题。 + - 末尾添加 `## Metadata` 部分，包含：keywords、persons、entities、topic。 + + 文件名和描述规则： + - `name`：简洁、稳定的主题/事件文件名 stem，采用 kebab-case。不含日期、不含 slash/反斜杠、不含 `< > : " | ? *`。 + - `description`：详尽的无损总结——具体到仅凭 description 就能传达全部核心信息。 + + ## 步骤 3 — 总结 + + 用一句话说明你做了什么（创建了哪个文件）。这是你最后一次文本输出。 + + ## 边界 + + - 当前 session 最多创建 3 条笔记。如果对话涵盖多个不同主题，将其拆分为独立笔记（每个主题一条）。如果只讨论了一个主题，则只创建一条笔记。 + +user_message_update: | + Today: {today} + Extra hint: {note} + Target path: {note_path} + + # Recent Conversation + + {history} + + # Your Task + + Extract all valuable information from the conversation above and merge them into the existing daily note at the target path. + + ## Step 1 — Skip Check + + Did the conversation produce substantive information worth long-term memory? Pure greetings or small talk → reply with a brief skip message and stop (do not call any tools). + + When truly ambiguous, default to writing — losing a memory is worse than writing one extra note. + + ## Step 2 — Read Existing Content + + **Call the `read` tool** with parameter `path` set to `{note_path}` to inspect the current note content. You MUST invoke it as a tool call, not as text. + - If the body is empty (only frontmatter, no actual content) → treat as new, jump to **Step 3b**. + - If there is body content → go to **Step 3a** to merge. + + ## Step 3a — Merge Update + + The note already has content. Extract new information from the conversation and merge into it. + + Extraction rules (apply to ALL new entries): + - Each new fact must be a complete, independent, self-contained statement (lossless restatement). + - PROHIBIT pronouns (he/she/it/they/this/that) — use full names and explicit references. + - PROHIBIT relative time (yesterday/today/last week) — use absolute dates (YYYY-MM-DD). + + Merge rules: + - **Timeline / history entries**: append only, never delete existing entries. + - **Current-state entries** (progress, blockers, next steps, open questions): rewrite the entire section to reflect the latest snapshot. + - **Everything else**: merge and deduplicate — keep all old facts, add new facts, remove exact duplicates. + - **Metadata section**: merge keywords, persons, entities, topics — union of old and new values. + + Execution: + 1. **Call the `edit` tool** for each section that needs updating, with parameters: `path`="{note_path}", `old`=, `new`=. You may call `edit` multiple times. Each call MUST be a tool call, not text. + 2. After body changes, **call the `frontmatter_update` tool** to refresh frontmatter, with parameters: `path`="{note_path}", `metadata`={{"name": "", "description": ""}}. + - Keep the existing `name` only when it is already the best concise topic/event filename stem. The system will rename the file after your final response. + - Do not add today's date or the daily directory date to `name`. + 3. If `edit` fails repeatedly (e.g., cannot find the original text due to formatting mismatch), fall back to **calling the `write` tool** with parameters: `path`="{note_path}", `name`=, `description`=, `content`= for a complete rewrite. + + ## Step 3b — Full Write (Empty File Fallback) + + The file exists but its body is empty. Extract and write the full content by **calling the `write` tool** with parameters: `path`="{note_path}", `name`=, `description`=, `content`=. + + Body requirements: + - Each factual entry must be a complete, independent, self-contained statement. + - PROHIBIT pronouns — use full names and explicit references. + - PROHIBIT relative time — use absolute dates (YYYY-MM-DD). + - Group entries by topic with headings. + - End with a `## Metadata` section containing: keywords, persons, entities, topic. + + Filename and description rules: + - `name`: concise, stable topic/event filename stem in kebab-case. No date. + - `description`: a thorough, lossless summary — the description alone should convey all key information. + + ## Step 4 — Summary + + State in one sentence what you did (what content was updated). This is your final text output. + + ## Boundaries + + - Only operate on one target path: `{note_path}`. Do not touch other notes. + - `write` unconditionally overwrites body and frontmatter — use with caution. + - Filename changes are expressed by updating frontmatter `name`. +user_message_update_zh: | + 今天：{today} + 额外提示：{note} + 目标路径：{note_path} + + # 最近的对话 + + {history} + + # 你的任务 + + 从上述对话中提取所有有价值的信息，并合并到目标路径的已有日记中。 + + ## 步骤 1 — 跳过检查 + + 对话是否产生了值得长期记忆的实质性信息？纯粹的寒暄或闲聊 → 回复一条简短的跳过消息并停止（不调用任何工具）。 + + 当真正模棱两可时，默认写入——丢失记忆比多写一条笔记更糟。 + + ## 步骤 2 — 读取现有内容 + + **调用 `read` 工具**，参数 `path` 设为 `{note_path}`，查看当前笔记内容。必须以工具调用方式调用，不要作为文本输出。 + - 如果正文为空（只有 frontmatter 无实际内容）→ 按新建处理，跳到 **步骤 3b**。 + - 如果有正文内容 → 转到 **步骤 3a** 进行合并。 + + ## 步骤 3a — 合并更新 + + 笔记已有内容。从对话中提取新信息并合并进去。 + + 提取规则（适用于所有新条目）： + - 每条新事实必须是完整的、独立的、自包含的陈述（无损复述）。 + - 禁止使用代词（他/她/它/他们/这个/那个）——必须使用全名和明确引用。 + - 禁止使用相对时间（昨天/今天/上周）——必须使用绝对日期（YYYY-MM-DD）。 + + 合并规则： + - **时间线 / 历史条目**：仅追加，永远不删除已有条目。 + - **当下状态类条目**（进度、卡点、下一步、未决问题）：整段重写，反映最新快照。 + - **其余内容**：合并去重——保留全部旧事实，添加新事实，去除完全重复项。 + - **Metadata 部分**：合并 keywords、persons、entities、topics——取旧值和新值的并集。 + + 执行： + 1. **调用 `edit` 工具**更新需要修改的部分，参数为：`path`="{note_path}"、`old`=<原文片段>、`new`=<替换片段>。可以多次调用 `edit`。每次调用必须是工具调用，不是文本。 + 2. 正文变更后，**调用 `frontmatter_update` 工具**刷新 frontmatter，参数为：`path`="{note_path}"、`metadata`={{"name": "<更新后的文件名 stem>", "description": "<更新后的总结>"}}。 + - 只有当前 `name` 已经是最合适的简洁主题/事件文件名 stem 时才保留。系统会在你最终回复后负责重命名文件。 + - 不要在 `name` 中加入今天日期或日记目录日期。 + 3. 如果 `edit` 多次失败（如因格式不匹配找不到原文），退回**调用 `write` 工具**全量重写，参数为：`path`="{note_path}"、`name`=、`description`=、`content`=<完整正文>。 + + ## 步骤 3b — 全量写入（空文件 fallback） + + 文件存在但正文为空。提取并**调用 `write` 工具**一次性写入完整内容，参数为：`path`="{note_path}"、`name`=、`description`=、`content`=<正文>。 + + 正文要求： + - 每条事实记录必须是完整的、独立的、自包含的陈述。 + - 禁止使用代词——必须使用全名和明确引用。 + - 禁止使用相对时间——必须使用绝对日期（YYYY-MM-DD）。 + - 按主题分组并使用标题。 + - 末尾添加 `## Metadata` 部分，包含：keywords、persons、entities、topic。 + + 文件名和描述规则： + - `name`：简洁、稳定的主题/事件文件名 stem，采用 kebab-case。不含日期。 + - `description`：详尽的无损总结——仅凭 description 就应能传达全部核心信息。 + + ## 步骤 4 — 总结 + + 用一句话说明你做了什么（更新了哪些内容）。这是你最后一次文本输出。 + + ## 边界 + + - 只针对一个目标路径：`{note_path}`。不要碰其他笔记。 + - `write` 会无条件覆盖正文和 frontmatter，请谨慎使用。 + - 文件名变化通过更新 frontmatter `name` 表达。 diff --git a/reme/steps/benchmark/lme/context_answer.py b/reme/steps/benchmark/lme/context_answer.py deleted file mode 100644 index 45134d8c..00000000 --- a/reme/steps/benchmark/lme/context_answer.py +++ /dev/null @@ -1,45 +0,0 @@ -"""Answer a query directly from the supplied session context.""" - -from ...base_step import BaseStep -from ....components import R - - -@R.register("context_answer_step") -class ContextAnswerStep(BaseStep): - """Answer a query using the LongMemEval direct-reading prompt.""" - - async def execute(self): - assert self.context is not None - query: str = self.context.get("query", "") - session_context: str = self.context.get("session_context", "") - current_date: str = self.context.get("current_date", "") - - if not query: - raise ValueError("context_answer_step requires non-empty query") - if not session_context: - raise ValueError("context_answer_step requires non-empty session_context") - if self.agent_wrapper is None: - raise ValueError("context_answer_step requires agent_wrapper") - - user_prompt = self.prompt_format( - "user_message", - session_context=session_context, - current_date=current_date, - query=query, - ) - result = await self.agent_wrapper.reply(user_prompt) - answer = (result.get("result") or "").strip() - - self.logger.info(f"[{self.name}] context answer: {answer}") - self.context["context_answer"] = answer - self.context.response.success = True - self.context.response.answer = answer - self.context.response.metadata.update( - { - "query": query, - "session_context": session_context, - "current_date": current_date, - "context_answer": answer, - }, - ) - return self.context.response diff --git a/reme/steps/benchmark/lme/context_answer.yaml b/reme/steps/benchmark/lme/context_answer.yaml deleted file mode 100644 index 207987b0..00000000 --- a/reme/steps/benchmark/lme/context_answer.yaml +++ /dev/null @@ -1,9 +0,0 @@ -user_message: | - I will give you several history chats between you and a user. Please answer the question - based on the relevant chat history. Answer the question step by step: first extract all the - relevant information, and then reason over the information to get the answer. - - History Chats: {session_context} - Current Date: {current_date} - Question: {query} - Answer (step by step): diff --git a/reme/steps/benchmark/lme/extract_session.py b/reme/steps/benchmark/lme/extract_session.py deleted file mode 100644 index 602a0d2f..00000000 --- a/reme/steps/benchmark/lme/extract_session.py +++ /dev/null @@ -1,103 +0,0 @@ -"""extract_session_by_id — deep-read one raw session, keyed by its session_id. - -This is the hand-written function tool the answering agent sees. Search results -surface a note's ``session_id``; when a hit looks relevant, the agent passes that -``session_id`` here. The step resolves the question/time from ``query.json``, -locates the raw session file (named ``_(...)_