mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-10-07 03:00:27 +00:00
feat(benchmark): add BEAM & restructure LongMemEval evaluation framework (#375)
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
* feat(eval): add LongMemEval evaluation framework with tool_defaults date injection
- Add evaluation/longmemeval/ with run.py, config.yaml, and test scripts
- Add reme/config/longmemeval.yaml for evaluation-specific model config
- Add tool_defaults mechanism to as_agent_wrapper for injecting default
tool kwargs (uses setdefault so LLM-provided values take priority)
- Pass tool_defaults={'daily_write': {'date': day}} in auto_memory to
ensure notes always use the correct historical date
- Add timestamp interpolation (_interpolate_timestamps) in auto_memory
for filling missing created_at fields via linear interpolation
- Evaluation pipeline: ingest sessions -> dream -> search -> answer -> judge
- Uses qwen3.6-flash for memory, qwen3.7-max for answer/judge
* chore: gitignore logs/results/demo.py, keep empty dirs
* chore: update .gitignore
* feat(eval): add multiprocessing and session time filtering to longmemeval runner
- Replace async execution with synchronous + multiprocessing for parallel item evaluation - Add filter_future_sessions option to only ingest sessions <= question date - Add question_types filtering in config - Add result summary with binary accuracy and avg score - Update config defaults (oracle variant, 50 items, 32 workers) - Minor code style fixes in agent_wrapper and auto_memory
* feat: add bench_query_step with ReAct agent for benchmark query phase
- Add BenchQueryStep using agent_wrapper with search job tool
- Replace manual search+LLM answer in run.py with bench_query_job
- Remove unused answer LLM config from longmemeval.yaml
- Register benchmark step module in steps/__init__.py
* feat: add start_date/end_date time filter support for search job
- Add _extract_date_from_path to extract validated YYYY-MM-DD from chunk paths
- Add start_date/end_date filtering in _matches_search_filter
- Implement progressive recall in FaissLocalFileStore.vector_search
- Promote start_date/end_date from context to search_filter in SearchStep
- Add start_date/end_date parameters to search job in default.yaml
- Add unit tests for date filter functionality
* fix: validate/normalize date filters and harden _extract_date_from_path
Address three code-review comments on the time_filter search feature:
1. Validate/normalize start_date and end_date before string comparison.
_matches_search_filter does lexicographic comparison against path_date
(always canonical YYYY-MM-DD). Raw caller values like '2026-2-28' or
'abc' would produce silently wrong results. Now SearchStep normalizes
valid dates via extract_daily_date (with strptime fallback for
non-zero-padded input) and silently ignores invalid dates with a
logger.warning, removing them from the filter.
2. Clarify behavior for paths without embedded dates.
Added optional strict_date_filter parameter (default False). When True
and at least one date bound is active, chunks whose path yields no date
(e.g. digest/personal/topic.md) are excluded. When False (default),
the existing behavior is preserved — dateless paths pass through.
3. Harden _extract_date_from_path against non-standard suffixes.
Previously parts[1].split('.')[0] accepted '2026-05-18.anything' as a
valid date. Now only exact 'YYYY-MM-DD' (dir) and 'YYYY-MM-DD.md'
(day-index) forms are accepted.
* feat(eval): LLM-as-Judge per-type prompt routing, binary-only, progress tracking
- Remove 0-5 score metric, keep only binary (yes/no) classification
- Load per-question-type judge prompts from llm-as-judge.json
(temporal-reasoning, knowledge-update, single-session-preference, __default__)
- Replace SCORE_JUDGE_PROMPT with type-specific BINARY_JUDGE_PROMPT template
- judge_response(): parameter 'metric' -> 'question_type', returns single 'judgment'
- Summary output: add per-type accuracy breakdown, remove score stats
- Add progress tracking: background thread prints PROGRESS every 10min
- Add FINAL progress line and total elapsed time on completion
- Add --log-level, --reme-log-level, -q CLI arguments
- Parallel mode: pool.map -> pool.imap_unordered for real-time progress
- config.yaml: full oracle (10000 items), 32 workers, all question types
- Add kill.sh (process cleanup) and run_async.sh (background eval launcher)
* docs: add LongMemEval oracle evaluation results (61.6% accuracy)
* feat(bench): add MAX_ITERATION limit to BenchQueryStep and add _auto_memory.yaml
* feat: add golden session benchmark & eval_only mode with refined prompt
- Add benchmark/longmemeval/run_golden_session.py for golden session evaluation
- Refine PROMPTED_SYSTEM_PROMPT: concise answer rule, remove 'Information not found' fallback
- Add eval_only mode to run.py (--eval_only flag)
- Add multiple eval config variants (evalonly, full, test5)
- Add analyze_results.py for result parsing
- Update auto_memory.yaml, longmemeval.yaml, application_config
- Update result-longmemeval.md with latest evaluation results
- Add benchmark results to .gitignore
* update: refine answer prompts and increase max iteration to 6 - Tighten prompted-answer system prompt for more concise output - Comment out 'Information not found' fallback rule - Increase MAX_ITERATION from 5 to 6 in bench_query - Add recall_eval.py - Update evaluation results
* feat(chunker): add dedicated JSON and JSONL file chunkers (cherry-pick from upstream #325)
- Add JsonFileChunker: structure-aware chunking preserving nested key paths,
optional list-to-dict conversion, size measured by json.dumps() char count
- Add JsonlFileChunker: line-aligned sliding-window chunking with configurable
overlap, supports char/byte mode switching
- Register both chunkers in default.yaml (json for .json, jsonl for .jsonl)
- Add comprehensive unit tests (21 + 20 test cases)
* feat(service): add CLI service for local job execution (from upstream #334)
- Introduce CliService to execute single jobs locally without serving ports
- Add prepare_start_config and should_precheck_start functions for CLI job setup
- Update reme start command to use CLI service when job argument is provided
- Add show_metadata to client kwargs for optional CLI metadata output
- Add unit tests for CLI service functionality and configuration handling
* feat(steps): add BM25/vector search steps, Python execute step, and draft steps (from upstream #334)
- Add Bm25SearchStep for plain BM25 keyword search with tool_context deduplication
- Add VectorSearchStep for plain vector search with tool_context deduplication
- Add PythonExecuteStep to run Python code in subprocess with timeout handling
- Add AddDraftStep/ReadAllDraftStep for draft accumulation scoped by tool context
- Update SearchStep with tool_context dedup, dynamic default limit via REME_SEARCH_LIMIT env,
and candidate_multiplier default changed from 3.0 to 5.0
- Add comprehensive unit tests for all new steps
* feat(search): add tool context deduplication and improve search configuration (#321)
* feat(search): add tool context deduplication and improve search configuration
- Modify _make_tool methods to accept and inject tool_context_id parameter
- Add tool_context_id handling in AS and CC agent wrappers
- Increase search candidate multiplier from 3.0 to 5.0 in default config
- Extend HTTP client timeout from 30s to 3600s
- Add tool context deduplication logic to prevent duplicate search results
- Implement TTL-based expiration for seen chunks in tool contexts
- Add comprehensive unit tests for tool context deduplication behavior
- Update .gitignore to exclude longmemeval directory
- Add time import for timestamp functionality in search step
* refactor(search): replace time module with datetime for timestamp generation
- Removed unused time import
- Added static method _now_ts using datetime.timestamp
- Updated clock parameter to use _now_ts method instead of time.time
- Maintained same timestamp precision and functionality
* fix(file_io): fix risk of out-workspace paths (#322)
* fix(file_io): fix risk of out-workspace paths
* chore(file_io): remove unused unittest file
* fix(as_embedding): support both agentscope 2.0.2 and 2.0.3 (#323)
2.0.3 promoted `dimensions` to a required first-class constructor
argument while keeping a backfill from `parameters.dimensions`; 2.0.2
has no such argument and reads `dimensions` from `Parameters`. Keep
`dimensions` in `Parameters` for both versions and, when the model
constructor accepts `dimensions`, pass `dimensions=None` so 2.0.3's
backfill promotes it out of `parameters`.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Bump version to 0.4.0.7
* refactor: delegate LLM-as-Judge to answer_judge_step and update eval config/results
- run.py: replace inline judge logic with judge_response_via_job using app.run_job('answer_judge')
- longmemeval.yaml: expand benchmark configuration
- bench_query.py: enhance benchmark query step
- result-longmemeval.md: update evaluation results
- judge_all_plus_results.json: add judge all-plus results
* refactor: split longmemeval.yaml into lme.yaml/beam.yaml and unify job names
- Split reme/config/longmemeval.yaml into lme.yaml (LongMemEval) and beam.yaml (BEAM)
- Unify job names across both configs: agentic_answer, answer_judge, context_answer
- Update evaluation/longmemeval/run.py and evaluation/beam/run_beam_eval.py to use unified job names
- Update all evaluation config YAMLs to reference lme.yaml
- Add BEAM benchmark step implementations (agentic_answer, context_answer, llm_judge)
- Remove obsolete config_test5.yaml and test_5sessions.py
* eval: BEAM 100K & LongMemEval cleaned-S 评测结果记录
- BEAM 100K eval-only (32并发, 20 case): Agentic 0.631, Prompted 0.468
- LongMemEval final GT (500题): Agentic 89.0%, Prompted 83.6%
- 新增 benchmark/result-beam.md, benchmark/result-longmemeval.md
- benchmark/beam/config.yaml: num_workers=32
* refactor: restructure benchmark directory and clean up gitignore rules
- Consolidate benchmark outputs to benchmark/results/ with .gitkeep
- Remove old benchmark scripts, configs and result files from benchmark/beam/ and benchmark/longmemeval/
- Add datasets/README.md and datasets/README_EN.md with download instructions
- Add datasets/longmemeval/download.py and final_groundtruth_cleaned_s.json
- Add memory_workspaces .gitkeep placeholders
- Restructure .gitignore: fix duplicate entries, add BEAM dataset exclusion, refine logs/results ignore patterns
- Remove stale result-beam.md and result-longmemeval.md from project root
* chore: clean up longmemeval benchmark scripts and update dataset docs
- Remove obsolete longmemeval benchmark runner/stats scripts
- Update datasets/longmemeval README and add Chinese translation
- Clean up final_groundtruth_cleaned_s.json
* docs(benchmark): add reproduction guide for LongMemEval and BEAM
- Add bilingual README for benchmark runners (EN/ZH)
- Cover prerequisites, dataset download, run commands, configs, outputs, logs, and kill.sh
* refactor: migrate auto_memory steps from evolve to benchmark-specific modules
- Split auto_memory into beam and lme benchmark-specific implementations
- Add auto_memory.py and auto_memory.yaml under steps/benchmark/beam and steps/benchmark/lme
- Slim down evolve/auto_memory.py and auto_memory.yaml to shared base only
- Remove obsolete evolve/_auto_memory.yaml
- Update benchmark run.py, config YAMLs, and step __init__.py registrations
- Update llm_judge and context_answer minor adjustments
- Remove outdated test_lme_final_answer_review.py
* revert(as_agent_wrapper): sync with upstream/main
Remove local-only comment to keep file identical with upstream/main.
* style: add trailing commas in benchmark __init__.py __all__ lists
* chore: disable vector_weight range assertion in SearchStep
* chore: add tests/integration/logs/ to .gitignore
* refactor: replace scipy.stats.kendalltau with pure numpy implementation
scipy is not listed in project dependencies. Implement Kendall's tau-b
rank correlation using only numpy to remove the undeclared dependency.
* feat(benchmark): add binary score metrics, update BEAM 1M results, and improve LLM retry/prompt config
- benchmark/beam/run.py: add binary score calculation per rubric item and per-type/overall binary stats
- benchmark/beam/config.yaml: switch to 1M dataset, reduce workers to 18
- benchmark/result-beam.md: add 1M evaluation results with binary scores
- benchmark/result-longmemeval.md: minor formatting
- reme/config/beam.yaml: increase max_retries to 5 and add retry_delay 5.0 for all LLM components
- reme/config/lme.yaml: increase max_retries to 5 and add retry_delay for judge/prompted/bench components
- reme/steps/benchmark/lme/agentic_answer.yaml: improve search strategy and answer rules prompts
* fix(benchmark): fix line-too-long and add pylint disable for main()
* refactor(longmemeval): use single cleaned-S dataset with embedded ground truth
- Switch to agentscope-ai/ReMe_longmemeval_clean_s_v2 HuggingFace source
- Remove separate final_groundtruth_cleaned_s.json (ground truth now in data file)
- Simplify download.py to fetch only longmemeval_s_reme_cleaned.json
- Remove dataset.variant and dataset.ground_truth_path config options
- Update benchmark and datasets READMEs to reflect new workflow
- Update .gitignore for new dataset filename
* fix: rename loop variable to avoid pylint redefined-outer-name warning
* refactor(benchmark): restructure datasets/memory_workspaces into benchmark and simplify auto_memory steps
* refactor(benchmark): extract BaseAgenticAnswerStep into base module
- Add reme/steps/benchmark/base/agentic_answer.py with shared agentic answer logic
- Refactor beam/lme AgenticAnswerStep to inherit from BaseAgenticAnswerStep
- Simplify lme/context_answer.py and update context_answer.yaml
- Update result-longmemeval.md with latest evaluation results (agentic 91.0%)
* refactor(benchmark): remove context_answer steps and unused configs
- Remove beam/lme context_answer job definitions and step implementations
- Remove prompted LLM component from beam.yaml and lme.yaml
- Delete jinli_lme.yaml (no longer needed)
- Simplify benchmark run.py scripts
- Clean up .gitkeep files and update .gitignore
- Remove unused import in search.py
* chore: remove benchmark/results/.gitkeep
---------
Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
Co-authored-by: jinliyl <6469360+jinliyl@users.noreply.github.com>
Co-authored-by: imrewce <wce@pku.edu.cn>
Co-authored-by: Sen Huang <48879559+ployts@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
xyf2020
GitHub
parent
e7d44f6f3b
commit
7b1da5a9ee
56 changed files with 5080 additions and 4834 deletions
18
.gitignore
vendored
18
.gitignore
vendored
|
|
@ -2,8 +2,8 @@
|
|||
.DS_Store
|
||||
.idea/
|
||||
.vscode/
|
||||
*.code-workspace
|
||||
.qoder/
|
||||
*.code-workspace
|
||||
|
||||
# Local environment
|
||||
.env
|
||||
|
|
@ -54,5 +54,17 @@ vault/
|
|||
docs/_build/
|
||||
site/
|
||||
|
||||
evaluation/
|
||||
datasets/
|
||||
# Memory workspaces (keep dirs, ignore contents)
|
||||
benchmark/memory_workspaces/
|
||||
|
||||
# LongMemEval raw data (download via download.py)
|
||||
benchmark/datasets/longmemeval/longmemeval_s_reme_cleaned.json
|
||||
|
||||
# BEAM dataset (clone via git)
|
||||
benchmark/datasets/BEAM/
|
||||
|
||||
# Benchmark outputs
|
||||
benchmark/results/
|
||||
|
||||
# integration tests outputs
|
||||
tests/integration/logs/
|
||||
|
|
|
|||
145
benchmark/README.md
Normal file
145
benchmark/README.md
Normal file
|
|
@ -0,0 +1,145 @@
|
|||
[中文版 / Chinese version](./README_ZH.md)
|
||||
|
||||
# ReMe Benchmarks
|
||||
|
||||
Reproduction guide for the two memory benchmarks shipped with ReMe:
|
||||
|
||||
- **LongMemEval** — long-term memory over multi-session chat histories.
|
||||
- **BEAM** — memory capability over long-context chat cases with rubric-based judging.
|
||||
|
||||
Each benchmark runs its own end-to-end pipeline: ingest sessions into an isolated
|
||||
per-item workspace, answer probing questions via an agentic (ReAct) mode,
|
||||
then score answers with an LLM-as-judge.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
Install ReMe with dev + core extras (Python 3.11+):
|
||||
|
||||
```bash
|
||||
pip install -e ".[dev,core]"
|
||||
```
|
||||
|
||||
Configure model credentials in a project-root `.env` file (copied from `example.env`).
|
||||
The runners auto-load `.env` from the repository root. Required variables typically include:
|
||||
|
||||
```bash
|
||||
LLM_API_KEY=...
|
||||
LLM_BASE_URL=...
|
||||
EMBEDDING_API_KEY=...
|
||||
EMBEDDING_BASE_URL=...
|
||||
```
|
||||
|
||||
Model names and component wiring live in the ReMe configs referenced by each benchmark
|
||||
(`reme/config/lme.yaml` and `reme/config/beam.yaml`).
|
||||
|
||||
## 2. Download Datasets
|
||||
|
||||
See [`datasets/README_EN.md`](datasets/README_EN.md) for full details.
|
||||
|
||||
**LongMemEval** (downloaded from a HuggingFace mirror):
|
||||
|
||||
```bash
|
||||
cd benchmark/datasets/longmemeval
|
||||
python download.py # downloads the cleaned-S dataset; skips if already present
|
||||
```
|
||||
|
||||
**BEAM** (public repository, cloned into `benchmark/datasets/`):
|
||||
|
||||
```bash
|
||||
cd benchmark/datasets
|
||||
git clone https://github.com/mohammadtavakoli78/BEAM.git
|
||||
```
|
||||
|
||||
## 3. Run LongMemEval
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
python benchmark/longmemeval/run.py
|
||||
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
|
||||
python benchmark/longmemeval/run.py -q # quiet: only eval-level logs
|
||||
python benchmark/longmemeval/run.py --log-level WARNING # reduce eval runner logs
|
||||
python benchmark/longmemeval/run.py --reme-log-level WARNING # reduce reme internal logs
|
||||
python benchmark/longmemeval/run.py --eval_only # reuse existing workspaces, query + judge only
|
||||
```
|
||||
|
||||
### Pipeline
|
||||
|
||||
1. Load the dataset (ground truth is embedded in the data file).
|
||||
2. For each item, create an isolated workspace and ingest sessions in chronological order.
|
||||
3. Trigger `auto_dream` when consecutive sessions cross the configured hour (default 23:00).
|
||||
4. Answer each question via agentic (ReAct) mode.
|
||||
5. Judge the answer (binary yes/no) with the `answer_judge` job and print per-type accuracy.
|
||||
|
||||
### Key config — `benchmark/longmemeval/config.yaml`
|
||||
|
||||
| Key | Meaning |
|
||||
| --- | --- |
|
||||
| `dataset.path` | Dataset file to evaluate (e.g. `longmemeval_s_reme_cleaned.json`); ground truth is included. |
|
||||
| `dataset.start_index` / `num_items` | Slice of items to evaluate. |
|
||||
| `dataset.question_types` | Filter by question type; empty = all. |
|
||||
| `dataset.workspace_root` | Per-item workspace root (`benchmark/memory_workspaces/longmemeval-s`). |
|
||||
| `evaluation.num_workers` | `0` = auto (cpu-2), `1` = sequential, `>1` = parallel. |
|
||||
| `evaluation.filter_future_sessions` | Only ingest sessions with timestamp ≤ `question_date`. |
|
||||
| `reme.config` | ReMe config used (`lme.yaml`). |
|
||||
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | Dream triggering behavior. |
|
||||
| `output.dir` | Results directory (`benchmark/results/longmemeval`). |
|
||||
|
||||
## 4. Run BEAM
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
python benchmark/beam/run.py
|
||||
python benchmark/beam/run.py --config benchmark/beam/config.yaml
|
||||
python benchmark/beam/run.py -q # quiet
|
||||
python benchmark/beam/run.py --eval_only # reuse existing workspaces, query + judge only
|
||||
```
|
||||
|
||||
### Pipeline
|
||||
|
||||
1. For each case, load `chat.json` and convert each batch into a ReMe session.
|
||||
2. Ingest sessions in chronological order into an isolated workspace, then `digest_update`.
|
||||
3. Answer each probing question via agentic (ReAct) mode.
|
||||
4. Score answers with BEAM's rubric-based `answer_judge` job and print per-type averages.
|
||||
|
||||
### Key config — `benchmark/beam/config.yaml`
|
||||
|
||||
| Key | Meaning |
|
||||
| --- | --- |
|
||||
| `dataset.beam_root` | BEAM dataset root (`benchmark/datasets/BEAM`). |
|
||||
| `dataset.chat_size` | Variant to run: `100K` / `500K` / `1M` / `10M`. |
|
||||
| `dataset.case_ids` | Specific cases (e.g. `["1","2"]`); empty = all cases. |
|
||||
| `dataset.start_index` / `num_items` | Case pagination (`num_items` `0` = all). |
|
||||
| `dataset.workspace_root` | Per-case workspace root (`benchmark/memory_workspaces/beam`). |
|
||||
| `evaluation.num_workers` | `0` = auto, `1` = sequential, `>1` = parallel. |
|
||||
| `reme.config` | ReMe config used (`beam.yaml`). |
|
||||
| `output.dir` | Results directory (`benchmark/results/beam`). |
|
||||
|
||||
## 5. Outputs & Logs
|
||||
|
||||
- **Results**: JSON files written to `output.dir`
|
||||
(`results_<timestamp>.json` for LongMemEval,
|
||||
`results_<chat_size>_<timestamp>.json` for BEAM). A summary with per-type
|
||||
accuracy/score is also printed to the console.
|
||||
- **Logs**: when `output.log_to_file` is enabled, per-run logs are written to
|
||||
`logs/<log_prefix>_<timestamp>/` (a `runner.log` plus one `worker-<pid>.log`
|
||||
per worker process).
|
||||
|
||||
## 6. Stopping a Run
|
||||
|
||||
Parallel runs spawn a process tree. To terminate a run and all its workers cleanly:
|
||||
|
||||
```bash
|
||||
bash benchmark/kill.sh <PID>
|
||||
```
|
||||
|
||||
The script gracefully sends `SIGTERM` to the whole process tree, then escalates to
|
||||
`SIGKILL` for any process that does not exit within 5 seconds.
|
||||
|
||||
## 7. Reference Results
|
||||
|
||||
Recorded evaluation results are available in:
|
||||
|
||||
- [`result-longmemeval.md`](./result-longmemeval.md)
|
||||
- [`result-beam.md`](./result-beam.md)
|
||||
142
benchmark/README_ZH.md
Normal file
142
benchmark/README_ZH.md
Normal file
|
|
@ -0,0 +1,142 @@
|
|||
# ReMe 评测复现说明
|
||||
|
||||
ReMe 内置两个记忆能力评测基准的复现指南：
|
||||
|
||||
- **LongMemEval** —— 面向多轮多会话历史的长期记忆能力评测。
|
||||
- **BEAM** —— 面向长上下文对话场景、基于评分细则（rubric）打分的记忆能力评测。
|
||||
|
||||
每个基准都会运行完整的端到端流程：将会话摄入独立的按条目隔离的工作区，
|
||||
以 agentic（ReAct）模式回答探测问题，
|
||||
最后由 LLM-as-judge 对答案进行打分。
|
||||
|
||||
## 1. 环境准备
|
||||
|
||||
安装 ReMe 及 dev + core 附加依赖（Python 3.11+）：
|
||||
|
||||
```bash
|
||||
pip install -e ".[dev,core]"
|
||||
```
|
||||
|
||||
在项目根目录配置 `.env`（可从 `example.env` 复制），运行脚本会自动从仓库根目录加载 `.env`。
|
||||
通常需要以下变量：
|
||||
|
||||
```bash
|
||||
LLM_API_KEY=...
|
||||
LLM_BASE_URL=...
|
||||
EMBEDDING_API_KEY=...
|
||||
EMBEDDING_BASE_URL=...
|
||||
```
|
||||
|
||||
模型名称与组件装配位于各基准引用的 ReMe 配置中
|
||||
（`reme/config/lme.yaml` 与 `reme/config/beam.yaml`）。
|
||||
|
||||
## 2. 下载数据集
|
||||
|
||||
完整说明见 [`datasets/README.md`](datasets/README.md)。
|
||||
|
||||
**LongMemEval**（从 HuggingFace 镜像下载）：
|
||||
|
||||
```bash
|
||||
cd benchmark/datasets/longmemeval
|
||||
python download.py # 下载 cleaned-S 数据文件，已存在则自动跳过
|
||||
```
|
||||
|
||||
**BEAM**（公开仓库，clone 到 `benchmark/datasets/` 下）：
|
||||
|
||||
```bash
|
||||
cd benchmark/datasets
|
||||
git clone https://github.com/mohammadtavakoli78/BEAM.git
|
||||
```
|
||||
|
||||
## 3. 运行 LongMemEval
|
||||
|
||||
在仓库根目录执行：
|
||||
|
||||
```bash
|
||||
python benchmark/longmemeval/run.py
|
||||
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
|
||||
python benchmark/longmemeval/run.py -q # 安静模式：仅评测级日志
|
||||
python benchmark/longmemeval/run.py --log-level WARNING # 降低评测 runner 日志
|
||||
python benchmark/longmemeval/run.py --reme-log-level WARNING # 降低 reme 内部日志
|
||||
python benchmark/longmemeval/run.py --eval_only # 复用已有工作区，仅执行查询 + 评判
|
||||
```
|
||||
|
||||
### 流程
|
||||
|
||||
1. 加载数据集（ground truth 已内嵌在数据文件中）。
|
||||
2. 为每个条目创建独立工作区，按时间顺序摄入会话。
|
||||
3. 当相邻会话跨越配置的时刻（默认 23:00）时触发 `auto_dream`。
|
||||
4. 以 agentic（ReAct）模式回答每个问题。
|
||||
5. 通过 `answer_judge` 任务对答案做二元（yes/no）评判，并输出各类型准确率。
|
||||
|
||||
### 关键配置 —— `benchmark/longmemeval/config.yaml`
|
||||
|
||||
| 配置项 | 含义 |
|
||||
| --- | --- |
|
||||
| `dataset.path` | 待评测的数据集文件（如 `longmemeval_s_reme_cleaned.json`），已包含 ground truth。 |
|
||||
| `dataset.start_index` / `num_items` | 评测条目的切片范围。 |
|
||||
| `dataset.question_types` | 按问题类型过滤，空表示全部。 |
|
||||
| `dataset.workspace_root` | 条目工作区根目录（`benchmark/memory_workspaces/longmemeval-s`）。 |
|
||||
| `evaluation.num_workers` | `0` = 自动（cpu-2），`1` = 串行，`>1` = 并行。 |
|
||||
| `evaluation.filter_future_sessions` | 仅摄入时间戳 ≤ `question_date` 的会话。 |
|
||||
| `reme.config` | 使用的 ReMe 配置（`lme.yaml`）。 |
|
||||
| `reme.dream_trigger_hour` / `dream_scan_days` / `dream_max_units` | dream 触发行为。 |
|
||||
| `output.dir` | 结果目录（`benchmark/results/longmemeval`）。 |
|
||||
|
||||
## 4. 运行 BEAM
|
||||
|
||||
在仓库根目录执行：
|
||||
|
||||
```bash
|
||||
python benchmark/beam/run.py
|
||||
python benchmark/beam/run.py --config benchmark/beam/config.yaml
|
||||
python benchmark/beam/run.py -q # 安静模式
|
||||
python benchmark/beam/run.py --eval_only # 复用已有工作区，仅执行查询 + 评判
|
||||
```
|
||||
|
||||
### 流程
|
||||
|
||||
1. 为每个 case 加载 `chat.json`，将每个 batch 转换为一个 ReMe 会话。
|
||||
2. 按时间顺序将会话摄入独立工作区，随后执行 `digest_update`。
|
||||
3. 以 agentic（ReAct）模式回答每个探测问题。
|
||||
4. 通过 BEAM 基于 rubric 的 `answer_judge` 任务打分，并输出各类型平均分。
|
||||
|
||||
### 关键配置 —— `benchmark/beam/config.yaml`
|
||||
|
||||
| 配置项 | 含义 |
|
||||
| --- | --- |
|
||||
| `dataset.beam_root` | BEAM 数据集根目录（`benchmark/datasets/BEAM`）。 |
|
||||
| `dataset.chat_size` | 运行的变体：`100K` / `500K` / `1M` / `10M`。 |
|
||||
| `dataset.case_ids` | 指定 case（如 `["1","2"]`），空表示全部。 |
|
||||
| `dataset.start_index` / `num_items` | case 分页（`num_items` 为 `0` 表示全部）。 |
|
||||
| `dataset.workspace_root` | case 工作区根目录（`benchmark/memory_workspaces/beam`）。 |
|
||||
| `evaluation.num_workers` | `0` = 自动，`1` = 串行，`>1` = 并行。 |
|
||||
| `reme.config` | 使用的 ReMe 配置（`beam.yaml`）。 |
|
||||
| `output.dir` | 结果目录（`benchmark/results/beam`）。 |
|
||||
|
||||
## 5. 输出与日志
|
||||
|
||||
- **结果**：JSON 文件写入 `output.dir`
|
||||
（LongMemEval 为 `results_<timestamp>.json`，
|
||||
BEAM 为 `results_<chat_size>_<timestamp>.json`）。同时控制台会打印含各类型
|
||||
准确率/分数的汇总。
|
||||
- **日志**：当 `output.log_to_file` 开启时，每次运行的日志写入
|
||||
`logs/<log_prefix>_<timestamp>/`（包含一个 `runner.log` 及每个 worker 进程的
|
||||
`worker-<pid>.log`）。
|
||||
|
||||
## 6. 终止运行
|
||||
|
||||
并行运行会派生进程树。若要干净地终止某次运行及其全部 worker：
|
||||
|
||||
```bash
|
||||
bash benchmark/kill.sh <PID>
|
||||
```
|
||||
|
||||
该脚本会先向整个进程树发送 `SIGTERM` 优雅终止，对 5 秒内未退出的进程再升级为 `SIGKILL`。
|
||||
|
||||
## 7. 参考结果
|
||||
|
||||
已记录的评测结果见：
|
||||
|
||||
- [`result-longmemeval.md`](./result-longmemeval.md)
|
||||
- [`result-beam.md`](./result-beam.md)
|
||||
23
benchmark/beam/config.yaml
Normal file
23
benchmark/beam/config.yaml
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
# BEAM evaluation configuration
|
||||
# This file controls what/how to evaluate.
|
||||
|
||||
dataset:
|
||||
beam_root: "benchmark/datasets/BEAM" # BEAM dataset root
|
||||
chat_size: "1M" # 100K | 500K | 1M | 10M (dataset variant)
|
||||
case_ids: [] # empty = all cases; or ["1", "2", "3"]
|
||||
start_index: 0 # first case index (for pagination)
|
||||
num_items: 0 # 0 = all cases; >0 = limit
|
||||
workspace_root: "benchmark/memory_workspaces/beam" # workspace root for case workspaces
|
||||
|
||||
evaluation:
|
||||
num_workers: 18 # 0 = auto; 1 = sequential; >1 = parallel (per-case)
|
||||
|
||||
reme:
|
||||
config: "beam.yaml" # reme config (in reme/config/)
|
||||
|
||||
output:
|
||||
dir: "benchmark/results/beam"
|
||||
log_dir: "logs" # log directory (relative to project root)
|
||||
log_prefix: "beam" # benchmark name used in log filenames
|
||||
log_to_console: true
|
||||
log_to_file: true
|
||||
791
benchmark/beam/run.py
Normal file
791
benchmark/beam/run.py
Normal file
|
|
@ -0,0 +1,791 @@
|
|||
"""BEAM evaluation runner for ReMe.
|
||||
|
||||
Evaluates ReMe's memory capability using the BEAM dataset.
|
||||
Each case gets an isolated workspace; chat.json batches are ingested as
|
||||
sessions in chronological order; finally probing questions are answered
|
||||
via an agentic (ReAct) approach, then
|
||||
judged by BEAM's rubric-based LLM-as-judge.
|
||||
|
||||
Usage:
|
||||
python benchmark/beam/run.py
|
||||
python benchmark/beam/run.py --config benchmark/beam/config.yaml
|
||||
python benchmark/beam/run.py -q # quiet: only eval-level logs
|
||||
python benchmark/beam/run.py --log-level WARNING # reduce eval runner logs
|
||||
python benchmark/beam/run.py --reme-log-level WARNING # reduce reme internal logs
|
||||
python benchmark/beam/run.py --eval_only # query+judge only, reuse existing workspace
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import time
|
||||
import threading
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# Load .env from project root
|
||||
_PROJECT_ROOT = Path(__file__).parent.parent.parent
|
||||
load_dotenv(_PROJECT_ROOT / ".env")
|
||||
|
||||
# Workspace root — read from config.yaml (dataset.workspace_root)
|
||||
_WORKSPACE_ROOT_DEFAULT = "benchmark/memory_workspaces/beam"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Logging
|
||||
# ---------------------------------------------------------------------------
|
||||
_DEFAULT_LOG_FORMAT = "%(asctime)s | %(levelname)s | %(message)s"
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format=_DEFAULT_LOG_FORMAT)
|
||||
logger = logging.getLogger("beam")
|
||||
|
||||
# Noisy library loggers silenced by default
|
||||
_NOISY_LOGGERS = [
|
||||
"httpx",
|
||||
"httpcore",
|
||||
"openai",
|
||||
"uvicorn",
|
||||
"multipart",
|
||||
"asyncio",
|
||||
"watchfiles",
|
||||
"filelock",
|
||||
]
|
||||
|
||||
|
||||
def setup_logging(
|
||||
log_level: str,
|
||||
reme_log_level: str,
|
||||
log_dir: str | None = None,
|
||||
):
|
||||
"""Configure logging for the eval runner and reme internals.
|
||||
|
||||
Args:
|
||||
log_level: Level for the eval runner logger (DEBUG/INFO/WARNING/ERROR).
|
||||
reme_log_level: Level for reme's internal loguru logger.
|
||||
log_dir: Per-run log directory (absolute path). None = no file logging.
|
||||
"""
|
||||
numeric = getattr(logging, log_level.upper(), logging.INFO)
|
||||
# Eval runner logger
|
||||
logging.getLogger().setLevel(numeric)
|
||||
logger.setLevel(numeric)
|
||||
|
||||
# Suppress noisy library loggers when above DEBUG
|
||||
if numeric > logging.DEBUG:
|
||||
for name in _NOISY_LOGGERS:
|
||||
lib_logger = logging.getLogger(name)
|
||||
lib_logger.setLevel(max(numeric, logging.WARNING))
|
||||
|
||||
# Add file handler for eval runner if log_dir is specified
|
||||
if log_dir:
|
||||
os.makedirs(log_dir, exist_ok=True)
|
||||
log_filepath = os.path.join(log_dir, "runner.log")
|
||||
file_handler = logging.FileHandler(log_filepath, encoding="utf-8")
|
||||
file_handler.setLevel(numeric)
|
||||
file_handler.setFormatter(logging.Formatter(_DEFAULT_LOG_FORMAT))
|
||||
logging.getLogger().addHandler(file_handler)
|
||||
logger.info(f"Eval runner log file: {log_filepath}")
|
||||
|
||||
# Reme internal logger (loguru) — will be applied per-worker via _configure_worker
|
||||
os.environ["REME_LOG_LEVEL"] = reme_log_level.upper()
|
||||
if log_dir:
|
||||
os.environ["REME_LOG_DIR"] = log_dir
|
||||
|
||||
|
||||
def _configure_worker(
|
||||
log_level: str,
|
||||
reme_log_level: str,
|
||||
log_dir: str | None = None,
|
||||
):
|
||||
"""Set up logging inside a multiprocessing worker process.
|
||||
|
||||
Must be called at the top of each worker because child processes inherit
|
||||
parent state but loguru sinks are NOT shared across fork/spawn.
|
||||
"""
|
||||
numeric = getattr(logging, log_level.upper(), logging.INFO)
|
||||
logging.basicConfig(level=numeric, format=_DEFAULT_LOG_FORMAT, force=True)
|
||||
logging.getLogger("beam").setLevel(numeric)
|
||||
if numeric > logging.DEBUG:
|
||||
for name in _NOISY_LOGGERS:
|
||||
logging.getLogger(name).setLevel(max(numeric, logging.WARNING))
|
||||
|
||||
# Add file handler for eval runner in worker process
|
||||
if log_dir:
|
||||
os.makedirs(log_dir, exist_ok=True)
|
||||
pid = os.getpid()
|
||||
log_filepath = os.path.join(log_dir, f"worker-{pid}.log")
|
||||
file_handler = logging.FileHandler(log_filepath, encoding="utf-8")
|
||||
file_handler.setLevel(numeric)
|
||||
file_handler.setFormatter(logging.Formatter(_DEFAULT_LOG_FORMAT))
|
||||
logging.getLogger().addHandler(file_handler)
|
||||
|
||||
# Re-initialize loguru for reme internals at the desired level
|
||||
from reme.utils import get_logger
|
||||
|
||||
reme_log_dir = log_dir or "logs"
|
||||
get_logger(log_dir=reme_log_dir, level=reme_log_level.upper(), force_init=True)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Config loading
|
||||
# ---------------------------------------------------------------------------
|
||||
def load_eval_config(config_path: str | None = None) -> dict:
|
||||
"""Load evaluation config yaml with env-var expansion."""
|
||||
if config_path is None:
|
||||
config_path = str(Path(__file__).parent / "config.yaml")
|
||||
with open(config_path, encoding="utf-8") as f:
|
||||
raw = f.read()
|
||||
|
||||
# Expand ${VAR} and ${VAR:-default}
|
||||
def _expand(m):
|
||||
expr = m.group(1)
|
||||
if ":-" in expr:
|
||||
key, default = expr.split(":-", 1)
|
||||
return os.environ.get(key, default)
|
||||
return os.environ.get(expr, "")
|
||||
|
||||
raw = re.sub(r"\$\{([^}]+)\}", _expand, raw)
|
||||
return yaml.safe_load(raw)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# BEAM data loading
|
||||
# ---------------------------------------------------------------------------
|
||||
def parse_beam_time_anchor(time_str: str) -> datetime:
|
||||
"""Parse BEAM time_anchor format: 'March-15-2024' -> datetime."""
|
||||
for fmt in ("%B-%d-%Y", "%b-%d-%Y"):
|
||||
try:
|
||||
return datetime.strptime(time_str, fmt)
|
||||
except ValueError:
|
||||
continue
|
||||
raise ValueError(f"Cannot parse time_anchor: {time_str!r}")
|
||||
|
||||
|
||||
def load_beam_chat(chat_path: Path, chat_size: str, case_id: str) -> list[dict]:
|
||||
"""Load BEAM chat.json and convert to ReMe session format.
|
||||
|
||||
Each batch becomes one session with all its turns flattened.
|
||||
Each turn resolves its own time_anchor independently; turns without
|
||||
an explicit time_anchor inherit from the most recent preceding turn.
|
||||
Returns list of sessions, each with:
|
||||
- session_id: str
|
||||
- date: str (YYYY-MM-DD) — derived from the *first* turn's time
|
||||
- messages: list[dict] with name, role, content, created_at
|
||||
"""
|
||||
with open(chat_path, encoding="utf-8") as f:
|
||||
batches = json.load(f)
|
||||
|
||||
sessions = []
|
||||
for batch in batches:
|
||||
batch_num = batch["batch_number"]
|
||||
|
||||
# Resolve batch-level fallback (used when no turn has a time_anchor)
|
||||
batch_anchor = batch.get("time_anchor")
|
||||
if not batch_anchor:
|
||||
batch_anchor = "January-1-2024"
|
||||
|
||||
# Flatten all turns, resolving time_anchor per turn
|
||||
messages = []
|
||||
prev_dt = None # carries forward from previous turn
|
||||
first_dt = None # for session-level date
|
||||
|
||||
for turn in batch["turns"]:
|
||||
# Find this turn's own time_anchor from its messages
|
||||
turn_anchor = None
|
||||
for msg in turn:
|
||||
if msg.get("time_anchor"):
|
||||
turn_anchor = msg["time_anchor"]
|
||||
break
|
||||
|
||||
if turn_anchor:
|
||||
dt = parse_beam_time_anchor(turn_anchor)
|
||||
elif prev_dt is not None:
|
||||
dt = prev_dt # inherit from previous turn
|
||||
else:
|
||||
dt = parse_beam_time_anchor(batch_anchor)
|
||||
|
||||
if first_dt is None:
|
||||
first_dt = dt
|
||||
prev_dt = dt
|
||||
|
||||
for msg in turn:
|
||||
role = msg["role"]
|
||||
messages.append(
|
||||
{
|
||||
"name": role,
|
||||
"role": role,
|
||||
"content": msg["content"],
|
||||
"created_at": dt.strftime("%Y-%m-%dT%H:%M:%S"),
|
||||
},
|
||||
)
|
||||
|
||||
sessions.append(
|
||||
{
|
||||
"session_id": f"beam_{chat_size}_{case_id}_batch{batch_num}",
|
||||
"date": first_dt.strftime("%Y-%m-%d"),
|
||||
"messages": messages,
|
||||
},
|
||||
)
|
||||
|
||||
return sessions
|
||||
|
||||
|
||||
def get_available_cases(beam_root: Path, chat_size: str) -> list[str]:
|
||||
"""Return sorted list of case IDs for a given chat size."""
|
||||
chats_dir = beam_root / "chats" / chat_size
|
||||
if not chats_dir.exists():
|
||||
return []
|
||||
return sorted(
|
||||
[d.name for d in chats_dir.iterdir() if d.is_dir()],
|
||||
key=int,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Answer generation
|
||||
# ---------------------------------------------------------------------------
|
||||
async def answer_question_agentic(app, question: str) -> tuple[str, dict]:
|
||||
"""Answer a probing question using ReMe's agentic_answer job.
|
||||
|
||||
Returns (answer, metadata)
|
||||
"""
|
||||
query_resp = await app.run_job(
|
||||
"agentic_answer",
|
||||
query=question,
|
||||
)
|
||||
answer = (query_resp.answer or "").strip()
|
||||
|
||||
return answer, {"mode": "agentic"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# BEAM rubric-based LLM-as-Judge
|
||||
# ---------------------------------------------------------------------------
|
||||
async def judge_answer(
|
||||
app,
|
||||
question: str,
|
||||
llm_response: str,
|
||||
rubric: list[str],
|
||||
question_type: str = "",
|
||||
) -> dict:
|
||||
"""Judge an answer via the answer_judge job (beam_rubric_judge_step)."""
|
||||
judge_resp = await app.run_job(
|
||||
"answer_judge",
|
||||
llm_response=llm_response,
|
||||
rubric=rubric,
|
||||
probing_question=question,
|
||||
question_type=question_type,
|
||||
)
|
||||
result = {
|
||||
"llm_judge_score": (judge_resp.metadata or {}).get("llm_judge_score", 0.0),
|
||||
"llm_judge_responses": (judge_resp.metadata or {}).get("llm_judge_responses", []),
|
||||
}
|
||||
# Include event_ordering extra metrics if present
|
||||
eo = (judge_resp.metadata or {}).get("event_ordering")
|
||||
if eo:
|
||||
result["event_ordering"] = eo
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main evaluation pipeline
|
||||
# ---------------------------------------------------------------------------
|
||||
async def evaluate_case(eval_config: dict, case_id: str, eval_only: bool = False) -> dict:
|
||||
"""Evaluate a single BEAM case end-to-end.
|
||||
|
||||
Args:
|
||||
eval_config: The evaluation configuration dict.
|
||||
case_id: The case directory name (e.g. "1").
|
||||
eval_only: If True, skip ingestion and only run query+judge
|
||||
using the existing workspace.
|
||||
|
||||
Returns:
|
||||
A results dict with all questions, answers, and judgments.
|
||||
"""
|
||||
from reme import Application
|
||||
from reme.config import resolve_app_config
|
||||
|
||||
dataset_cfg = eval_config["dataset"]
|
||||
chat_size = dataset_cfg["chat_size"]
|
||||
beam_root = _PROJECT_ROOT / dataset_cfg.get("beam_root", "benchmark/datasets/BEAM")
|
||||
chat_path = beam_root / "chats" / chat_size / case_id / "chat.json"
|
||||
probing_questions_path = beam_root / "chats" / chat_size / case_id / "probing_questions" / "probing_questions.json"
|
||||
|
||||
if not chat_path.exists():
|
||||
raise FileNotFoundError(f"Chat file not found: {chat_path}")
|
||||
if not probing_questions_path.exists():
|
||||
raise FileNotFoundError(f"Probing questions not found: {probing_questions_path}")
|
||||
|
||||
logger.info(
|
||||
"[Case %s] size=%s%s",
|
||||
case_id,
|
||||
chat_size,
|
||||
" [eval_only]" if eval_only else "",
|
||||
)
|
||||
|
||||
# Workspace setup
|
||||
workspace_root = _PROJECT_ROOT / dataset_cfg.get("workspace_root", _WORKSPACE_ROOT_DEFAULT)
|
||||
case_dir = workspace_root / f"{chat_size}_{case_id}"
|
||||
workspace_dir = str(case_dir / ".reme")
|
||||
|
||||
if eval_only:
|
||||
if not case_dir.exists() or not Path(workspace_dir).exists():
|
||||
raise FileNotFoundError(
|
||||
f"[Case {case_id}] eval_only: workspace not found at {case_dir}. "
|
||||
f"Run without --eval_only first to build the workspace.",
|
||||
)
|
||||
else:
|
||||
if case_dir.exists():
|
||||
shutil.rmtree(case_dir)
|
||||
logger.info(f"[Case {case_id}] Cleaned existing workspace: {case_dir}")
|
||||
else:
|
||||
logger.info(f"[Case {case_id}] Workspace not found, creating: {case_dir}")
|
||||
case_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Pre-initialize ReMe's loguru logger with the correct log_dir
|
||||
output_cfg = eval_config.get("output", {})
|
||||
if output_cfg.get("log_to_file", False):
|
||||
reme_log_dir = os.environ.get("REME_LOG_DIR")
|
||||
if reme_log_dir:
|
||||
from reme.utils import get_logger
|
||||
|
||||
get_logger(
|
||||
log_dir=reme_log_dir,
|
||||
level=os.environ.get("REME_LOG_LEVEL", "INFO"),
|
||||
log_to_console=output_cfg.get("log_to_console", True),
|
||||
log_to_file=True,
|
||||
force_init=True,
|
||||
)
|
||||
|
||||
cfg = resolve_app_config(
|
||||
config=eval_config["reme"]["config"],
|
||||
workspace_dir=workspace_dir,
|
||||
log_to_console=output_cfg.get("log_to_console", True),
|
||||
log_to_file=output_cfg.get("log_to_file", False),
|
||||
enable_logo=False,
|
||||
)
|
||||
|
||||
app = Application(**cfg)
|
||||
await app.start()
|
||||
|
||||
sessions_ingested = 0
|
||||
try:
|
||||
if not eval_only:
|
||||
# ── Phase 1: Ingest sessions ──────────────────────────────
|
||||
sessions = load_beam_chat(chat_path, chat_size, case_id)
|
||||
logger.info(f"[Case {case_id}] Loaded {len(sessions)} sessions from chat.json")
|
||||
|
||||
for i, session in enumerate(sessions):
|
||||
logger.info(
|
||||
f"[Case {case_id}] Ingesting session {i+1}/{len(sessions)}: "
|
||||
f"id={session['session_id']} date={session['date']} "
|
||||
f"msgs={len(session['messages'])}",
|
||||
)
|
||||
resp = await app.run_job(
|
||||
"auto_memory",
|
||||
messages=session["messages"],
|
||||
session_id=session["session_id"],
|
||||
date=session["date"],
|
||||
)
|
||||
if not resp.success:
|
||||
logger.warning(f"[Case {case_id}] auto_memory failed: {resp.answer}")
|
||||
else:
|
||||
logger.info(
|
||||
f"[Case {case_id}] auto_memory success: " f"{resp.answer[:100] if resp.answer else ''}",
|
||||
)
|
||||
await app.run_job("index_update")
|
||||
sessions_ingested += 1
|
||||
|
||||
# Final digest update
|
||||
logger.info(f"[Case {case_id}] Running digest_update...")
|
||||
await app.run_job("digest_update")
|
||||
logger.info(f"[Case {case_id}] Ingestion complete.")
|
||||
|
||||
# ── Phase 2: Answer + Judge probing questions ───────────────
|
||||
with open(probing_questions_path, encoding="utf-8") as f:
|
||||
probing_questions = json.load(f)
|
||||
|
||||
total_questions = sum(len(v) for v in probing_questions.values())
|
||||
logger.info(f"[Case {case_id}] Total probing questions: {total_questions}")
|
||||
|
||||
all_question_results = []
|
||||
q_idx = 0
|
||||
|
||||
for q_type in probing_questions:
|
||||
logger.info(
|
||||
f"[Case {case_id}] Question type: {q_type} " f"({len(probing_questions[q_type])} questions)",
|
||||
)
|
||||
|
||||
for i, q in enumerate(probing_questions[q_type]):
|
||||
q_idx += 1
|
||||
question = q["question"]
|
||||
rubric = q.get("rubric", [])
|
||||
logger.info(
|
||||
f"[Case {case_id}] [{q_idx}/{total_questions}] " f"{q_type} Q{i+1}: {question[:100]}...",
|
||||
)
|
||||
|
||||
q_result = {
|
||||
"question_type": q_type,
|
||||
"question_index": i,
|
||||
"question": question,
|
||||
"rubric": rubric,
|
||||
}
|
||||
|
||||
# Agentic answer
|
||||
try:
|
||||
agentic_answer, agentic_meta = await answer_question_agentic(
|
||||
app,
|
||||
question,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"[Case {case_id}] Agentic answer failed: {e}")
|
||||
agentic_answer = f"(error: {e})"
|
||||
agentic_meta = {"error": str(e)}
|
||||
|
||||
if not agentic_answer:
|
||||
agentic_answer = "(no answer generated)"
|
||||
logger.info(f"[Case {case_id}] Agentic answer: {agentic_answer[:200]}...")
|
||||
|
||||
# Judge agentic answer
|
||||
logger.info(f"[Case {case_id}] Judging agentic ({q_type})...")
|
||||
agentic_judgment = await judge_answer(
|
||||
app,
|
||||
question,
|
||||
agentic_answer,
|
||||
rubric,
|
||||
question_type=q_type,
|
||||
)
|
||||
logger.info(
|
||||
f"[Case {case_id}] Agentic score: " f"{agentic_judgment['llm_judge_score']:.3f}",
|
||||
)
|
||||
|
||||
q_result["agentic_response"] = agentic_answer
|
||||
q_result["agentic_judgment"] = agentic_judgment
|
||||
q_result["agentic_metadata"] = agentic_meta
|
||||
|
||||
all_question_results.append(q_result)
|
||||
|
||||
finally:
|
||||
await app.close()
|
||||
|
||||
return {
|
||||
"case_id": case_id,
|
||||
"chat_size": chat_size,
|
||||
"sessions_ingested": sessions_ingested,
|
||||
"total_questions": len(all_question_results),
|
||||
"questions": all_question_results,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Worker: runs a single case in its own process with its own event loop
|
||||
# ---------------------------------------------------------------------------
|
||||
def _evaluate_case_worker(task_input: tuple) -> dict:
|
||||
"""Worker function for multiprocessing. Each process gets its own event loop."""
|
||||
eval_config, case_id, log_level, reme_log_level, eval_only, log_dir = task_input
|
||||
import asyncio # pylint: disable=import-outside-toplevel
|
||||
|
||||
_configure_worker(log_level, reme_log_level, log_dir=log_dir)
|
||||
|
||||
# Suppress httpx GC noise
|
||||
logging.getLogger("asyncio").setLevel(logging.CRITICAL)
|
||||
|
||||
return asyncio.run(evaluate_case(eval_config, case_id, eval_only=eval_only))
|
||||
|
||||
|
||||
def _indexed_worker(indexed_input: tuple) -> tuple:
|
||||
"""Module-level wrapper for imap_unordered with index tracking."""
|
||||
idx, task_input = indexed_input
|
||||
return idx, _evaluate_case_worker(task_input)
|
||||
|
||||
|
||||
def _resolve_num_workers(configured: int) -> int:
|
||||
"""Resolve num_workers: 0=auto (cpu_count-2, min 1), 1=sequential, >1=parallel."""
|
||||
if configured == 0:
|
||||
return max(1, (os.cpu_count() or 4) - 2)
|
||||
return max(1, configured)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Entry point
|
||||
# ---------------------------------------------------------------------------
|
||||
def main( # pylint: disable=too-many-statements
|
||||
config_path: str | None = None,
|
||||
log_level: str = "INFO",
|
||||
reme_log_level: str = "INFO",
|
||||
eval_only: bool = False,
|
||||
):
|
||||
"""Run the BEAM evaluation pipeline.
|
||||
|
||||
Args:
|
||||
config_path: Path to the YAML config file.
|
||||
log_level: Log level for the eval runner.
|
||||
reme_log_level: Log level for reme internal logs.
|
||||
eval_only: If True, skip ingestion and only run query+judge using
|
||||
existing workspaces.
|
||||
"""
|
||||
from multiprocessing import Pool # pylint: disable=import-outside-toplevel
|
||||
|
||||
# Load config BEFORE logging setup so log_dir is available
|
||||
eval_config = load_eval_config(config_path)
|
||||
|
||||
# Resolve per-run log directory from config
|
||||
output_cfg = eval_config.get("output", {})
|
||||
log_dir_abs = None
|
||||
if output_cfg.get("log_to_file", False):
|
||||
log_dir_raw = output_cfg.get("log_dir", "logs")
|
||||
log_prefix = output_cfg.get("log_prefix", "beam")
|
||||
run_ts = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
|
||||
log_dir_abs = str(_PROJECT_ROOT / log_dir_raw / f"{log_prefix}_{run_ts}")
|
||||
|
||||
setup_logging(log_level, reme_log_level, log_dir=log_dir_abs)
|
||||
dataset_cfg = eval_config["dataset"]
|
||||
chat_size = dataset_cfg["chat_size"]
|
||||
beam_root = _PROJECT_ROOT / dataset_cfg.get("beam_root", "benchmark/datasets/BEAM")
|
||||
|
||||
# Determine which cases to run
|
||||
case_ids = dataset_cfg.get("case_ids") or []
|
||||
if not case_ids:
|
||||
case_ids = get_available_cases(beam_root, chat_size)
|
||||
|
||||
# Pagination
|
||||
start = dataset_cfg.get("start_index", 0)
|
||||
num_items = dataset_cfg.get("num_items", 0)
|
||||
if num_items > 0:
|
||||
case_ids = case_ids[start : start + num_items]
|
||||
elif start > 0:
|
||||
case_ids = case_ids[start:]
|
||||
|
||||
if not case_ids:
|
||||
logger.error(f"No cases found for chat_size={chat_size}")
|
||||
return
|
||||
|
||||
logger.info(
|
||||
"Evaluating %d case(s) for chat_size=%s: %s%s",
|
||||
len(case_ids),
|
||||
chat_size,
|
||||
case_ids,
|
||||
" [eval_only: query+judge only]" if eval_only else "",
|
||||
)
|
||||
|
||||
# Resolve parallelism
|
||||
num_workers = _resolve_num_workers(eval_config["evaluation"].get("num_workers", 1))
|
||||
logger.info(f"Using {num_workers} worker(s)")
|
||||
|
||||
# Create output directory
|
||||
output_dir = _PROJECT_ROOT / output_cfg.get("dir", "benchmark/results/beam")
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Create workspace root directory
|
||||
workspace_root = _PROJECT_ROOT / dataset_cfg.get("workspace_root", _WORKSPACE_ROOT_DEFAULT)
|
||||
workspace_root.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Pre-check: verify all workspaces exist in eval_only mode
|
||||
if eval_only:
|
||||
missing_cases = []
|
||||
for case_id in case_ids:
|
||||
case_dir = workspace_root / f"{chat_size}_{case_id}"
|
||||
if not case_dir.exists() or not (case_dir / ".reme").exists():
|
||||
missing_cases.append(case_id)
|
||||
if missing_cases:
|
||||
preview = missing_cases[:10]
|
||||
suffix = "..." if len(missing_cases) > 10 else ""
|
||||
raise FileNotFoundError(
|
||||
f"eval_only: {len(missing_cases)} workspace(s) not found under {workspace_root}. "
|
||||
f"Missing cases: {preview}{suffix}. "
|
||||
f"Run without --eval_only first to build the workspaces.",
|
||||
)
|
||||
|
||||
# Build task args
|
||||
task_args = [(eval_config, case_id, log_level, reme_log_level, eval_only, log_dir_abs) for case_id in case_ids]
|
||||
|
||||
# Progress tracking
|
||||
total_items = len(task_args)
|
||||
completed_count = [0]
|
||||
start_time = time.time()
|
||||
progress_lock = threading.Lock()
|
||||
|
||||
def _print_progress(prefix: str = "PROGRESS"):
|
||||
elapsed = time.time() - start_time
|
||||
elapsed_min = elapsed / 60
|
||||
done = completed_count[0]
|
||||
pct = 100.0 * done / total_items if total_items else 0
|
||||
eta_str = "N/A"
|
||||
if done > 0:
|
||||
eta_sec = elapsed / done * (total_items - done)
|
||||
eta_str = f"{eta_sec/60:.1f}min"
|
||||
print(
|
||||
f"[{prefix}] {datetime.now().strftime('%Y-%m-%d %H:%M:%S')} | "
|
||||
f"{done}/{total_items} ({pct:.1f}%) completed | "
|
||||
f"elapsed={elapsed_min:.1f}min | ETA={eta_str}",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
def _progress_timer():
|
||||
"""Background thread: print progress every 10 minutes."""
|
||||
while not _timer_stop.is_set():
|
||||
_timer_stop.wait(600)
|
||||
if not _timer_stop.is_set():
|
||||
with progress_lock:
|
||||
_print_progress()
|
||||
|
||||
_timer_stop = threading.Event()
|
||||
timer_thread = threading.Thread(target=_progress_timer, daemon=True)
|
||||
timer_thread.start()
|
||||
|
||||
# Run evaluation
|
||||
if num_workers == 1:
|
||||
results = []
|
||||
for task_input in task_args:
|
||||
result = _evaluate_case_worker(task_input)
|
||||
results.append(result)
|
||||
with progress_lock:
|
||||
completed_count[0] += 1
|
||||
else:
|
||||
results = [None] * total_items
|
||||
indexed_args = list(enumerate(task_args))
|
||||
|
||||
with Pool(processes=num_workers) as pool:
|
||||
for idx, result in pool.imap_unordered(_indexed_worker, indexed_args):
|
||||
results[idx] = result
|
||||
with progress_lock:
|
||||
completed_count[0] += 1
|
||||
|
||||
# Stop progress timer
|
||||
_timer_stop.set()
|
||||
timer_thread.join(timeout=2)
|
||||
|
||||
# Save results
|
||||
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
output_file = output_dir / f"results_{chat_size}_{timestamp}.json"
|
||||
with open(output_file, "w", encoding="utf-8") as f:
|
||||
json.dump(results, f, ensure_ascii=False, indent=2)
|
||||
logger.info(f"Results saved to {output_file}")
|
||||
|
||||
# Final progress
|
||||
_print_progress("FINAL")
|
||||
|
||||
# Print concise summary
|
||||
print("\n" + "=" * 70)
|
||||
print(f" BEAM EVALUATION RESULTS | size={chat_size} cases={len(results)}")
|
||||
print("=" * 70)
|
||||
|
||||
# Per-type stats (agentic only)
|
||||
type_scores: dict[str, list[float]] = {}
|
||||
type_binary_scores: dict[str, list[float]] = {}
|
||||
all_scores: list[float] = []
|
||||
all_binary_scores: list[float] = []
|
||||
|
||||
for case_result in results:
|
||||
if "error" in case_result:
|
||||
continue
|
||||
for q in case_result.get("questions", []):
|
||||
judgment = q.get("agentic_judgment", {})
|
||||
score = judgment.get("llm_judge_score", 0.0)
|
||||
# Binary: convert each rubric item score to 0/1, then average
|
||||
judge_responses = judgment.get("llm_judge_responses", [])
|
||||
if judge_responses:
|
||||
binary_scores_per_item = [1.0 if r.get("score", 0) >= 1.0 else 0.0 for r in judge_responses]
|
||||
binary_score = sum(binary_scores_per_item) / len(binary_scores_per_item)
|
||||
else:
|
||||
binary_score = 1.0 if score > 0.99 else 0.0
|
||||
qtype = q["question_type"]
|
||||
if qtype not in type_scores:
|
||||
type_scores[qtype] = []
|
||||
type_binary_scores[qtype] = []
|
||||
type_scores[qtype].append(score)
|
||||
type_binary_scores[qtype].append(binary_score)
|
||||
all_scores.append(score)
|
||||
all_binary_scores.append(binary_score)
|
||||
|
||||
print("\n ── AGENTIC ──")
|
||||
if all_scores:
|
||||
for qtype in sorted(type_scores.keys()):
|
||||
scores = type_scores[qtype]
|
||||
avg = sum(scores) / len(scores) if scores else 0
|
||||
bin_scores = type_binary_scores[qtype]
|
||||
bin_avg = sum(bin_scores) / len(bin_scores) if bin_scores else 0
|
||||
print(f" {qtype:<40s}: {avg:.3f} binary={bin_avg:.3f} ({len(scores)} Qs)")
|
||||
overall = sum(all_scores) / len(all_scores) if all_scores else 0
|
||||
binary_overall = sum(all_binary_scores) / len(all_binary_scores) if all_binary_scores else 0
|
||||
print(f" {'-'*38}")
|
||||
print(f" {'OVERALL':<40s}: {overall:.3f} binary={binary_overall:.3f} ({len(all_scores)} Qs)")
|
||||
else:
|
||||
print(" (no results)")
|
||||
|
||||
# Per-case summary
|
||||
print("\n ── Per-Case Summary ──")
|
||||
for case_result in results:
|
||||
case_id = case_result["case_id"]
|
||||
if "error" in case_result:
|
||||
print(f" Case {case_id}: ERROR — {case_result['error']}")
|
||||
continue
|
||||
n_qs = case_result.get("total_questions", 0)
|
||||
n_sessions = case_result.get("sessions_ingested", 0)
|
||||
parts = [f"Case {case_id}: {n_sessions} sessions, {n_qs} questions"]
|
||||
questions = case_result.get("questions", [])
|
||||
scores = [q.get("agentic_judgment", {}).get("llm_judge_score", 0.0) for q in questions]
|
||||
if scores:
|
||||
avg = sum(scores) / len(scores)
|
||||
# Binary: 0/1 per rubric item, average per question, then across questions
|
||||
bin_scores = []
|
||||
for q in questions:
|
||||
judge_responses = q.get("agentic_judgment", {}).get("llm_judge_responses", [])
|
||||
if judge_responses:
|
||||
item_bins = [1.0 if r.get("score", 0) >= 1.0 else 0.0 for r in judge_responses]
|
||||
bin_scores.append(sum(item_bins) / len(item_bins))
|
||||
else:
|
||||
s = q.get("agentic_judgment", {}).get("llm_judge_score", 0.0)
|
||||
bin_scores.append(1.0 if s > 0.99 else 0.0)
|
||||
bin_avg = sum(bin_scores) / len(bin_scores)
|
||||
parts.append(f"agentic={avg:.3f} binary={bin_avg:.3f}")
|
||||
print(f" {' | '.join(parts)}")
|
||||
|
||||
print("=" * 70)
|
||||
total_elapsed = time.time() - start_time
|
||||
print(f"\n Total time: {total_elapsed/60:.1f} min")
|
||||
print("\n" + "=" * 70)
|
||||
print(" [DONE] BEAM EVALUATION COMPLETED SUCCESSFULLY")
|
||||
print("=" * 70 + "\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(description="BEAM evaluation runner")
|
||||
parser.add_argument("--config", type=str, default=None, help="Path to config.yaml")
|
||||
parser.add_argument(
|
||||
"--log-level",
|
||||
type=str,
|
||||
default="INFO",
|
||||
choices=["DEBUG", "INFO", "WARNING", "ERROR"],
|
||||
help="Log level for the eval runner (default: INFO)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--reme-log-level",
|
||||
type=str,
|
||||
default="INFO",
|
||||
choices=["DEBUG", "INFO", "WARNING", "ERROR"],
|
||||
help="Log level for reme internal logs — loguru (default: INFO)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-q",
|
||||
"--quiet",
|
||||
action="store_true",
|
||||
help="Shortcut for --log-level WARNING --reme-log-level WARNING",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--eval_only",
|
||||
action="store_true",
|
||||
help="Skip ingestion. Reuse existing workspaces and only run query+judge.",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.quiet:
|
||||
args.log_level = "WARNING"
|
||||
args.reme_log_level = "WARNING"
|
||||
|
||||
main(args.config, args.log_level, args.reme_log_level, eval_only=args.eval_only)
|
||||
40
benchmark/datasets/README.md
Normal file
40
benchmark/datasets/README.md
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
# 数据集下载说明
|
||||
|
||||
本目录包含 ReMe 评测所需的数据集。部分数据集体积较大，不纳入 Git 版本管理，需要手动下载。
|
||||
|
||||
## LongMemEval（cleaned-S）
|
||||
|
||||
ReMe 仅使用 LongMemEval 的 **cleaned-S** 版本，数据托管在 HuggingFace：
|
||||
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)
|
||||
（下载脚本经 hf-mirror.com 镜像源获取）。
|
||||
|
||||
按以下步骤下载：
|
||||
|
||||
```bash
|
||||
cd benchmark/datasets/longmemeval
|
||||
|
||||
# 下载 cleaned-S 数据文件（已存在则自动跳过）
|
||||
python download.py
|
||||
```
|
||||
|
||||
下载完成后，目录下应包含以下文件：
|
||||
|
||||
| 文件名 | 说明 |
|
||||
| --- | --- |
|
||||
| `longmemeval_s_reme_cleaned.json` | cleaned-S 数据集，已包含 ground truth 字段 |
|
||||
| `download.py` | 下载脚本（已随仓库提供） |
|
||||
|
||||
> **注意**：下载脚本使用 hf-mirror.com 镜像源，如需更换源请修改 `download.py` 中的 `BASE_URL`。
|
||||
|
||||
下载完成后即可参照 [`benchmark/README_ZH.md`](../README_ZH.md) 运行 LongMemEval 评测。
|
||||
|
||||
## BEAM
|
||||
|
||||
BEAM 数据集为公开仓库，直接 clone 到 `benchmark/datasets/` 目录下即可：
|
||||
|
||||
```bash
|
||||
cd benchmark/datasets
|
||||
git clone https://github.com/mohammadtavakoli78/BEAM.git
|
||||
```
|
||||
|
||||
clone 完成后，`benchmark/datasets/BEAM/` 目录下应包含 `chats/`、`src/`、`topics/` 等子目录。
|
||||
40
benchmark/datasets/README_EN.md
Normal file
40
benchmark/datasets/README_EN.md
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
# Dataset Download Guide
|
||||
|
||||
This directory contains datasets required for ReMe evaluation. Some datasets are large and excluded from Git version control — they must be downloaded manually.
|
||||
|
||||
## LongMemEval (cleaned-S)
|
||||
|
||||
ReMe uses only the **cleaned-S** split of LongMemEval, hosted on HuggingFace:
|
||||
[agentscope-ai/ReMe_longmemeval_clean_s_v2](https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2)
|
||||
(the script downloads via the hf-mirror.com mirror).
|
||||
|
||||
Download it with:
|
||||
|
||||
```bash
|
||||
cd benchmark/datasets/longmemeval
|
||||
|
||||
# Download the cleaned-S data file (skipped automatically if it already exists)
|
||||
python download.py
|
||||
```
|
||||
|
||||
After downloading, the directory should contain:
|
||||
|
||||
| File | Description |
|
||||
| --- | --- |
|
||||
| `longmemeval_s_reme_cleaned.json` | cleaned-S dataset with ground truth fields included |
|
||||
| `download.py` | Download script (included in repo) |
|
||||
|
||||
> **Note**: The download script uses hf-mirror.com by default. To use a different mirror, modify `BASE_URL` in `download.py`.
|
||||
|
||||
Once the download completes, follow [`benchmark/README.md`](../README.md) to run the LongMemEval evaluation.
|
||||
|
||||
## BEAM
|
||||
|
||||
BEAM is a public repository. Clone it directly into the `benchmark/datasets/` directory:
|
||||
|
||||
```bash
|
||||
cd benchmark/datasets
|
||||
git clone https://github.com/mohammadtavakoli78/BEAM.git
|
||||
```
|
||||
|
||||
After cloning, `benchmark/datasets/BEAM/` should contain `chats/`, `src/`, `topics/` and other subdirectories.
|
||||
65
benchmark/datasets/longmemeval/download.py
Normal file
65
benchmark/datasets/longmemeval/download.py
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
"""Download the LongMemEval cleaned-S dataset used by ReMe.
|
||||
|
||||
Source: https://huggingface.co/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2
|
||||
(downloaded via the hf-mirror.com mirror for reliability).
|
||||
|
||||
The file ``longmemeval_s_reme_cleaned.json`` is saved under this directory using the same
|
||||
name as on the remote (``benchmark/longmemeval/config.yaml`` points to it).
|
||||
|
||||
Usage:
|
||||
python download.py # download cleaned-S (skip if it already exists)
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import urllib.request
|
||||
|
||||
BASE_URL = "https://hf-mirror.com/datasets/agentscope-ai/ReMe_longmemeval_clean_s_v2/resolve/main"
|
||||
TARGET_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
|
||||
# Files to download (saved with the same name as on the remote).
|
||||
FILES = [
|
||||
"longmemeval_s_reme_cleaned.json",
|
||||
]
|
||||
|
||||
|
||||
def download_file(filename: str):
|
||||
"""Download a single file from the mirror to the target directory."""
|
||||
url = f"{BASE_URL}/{filename}"
|
||||
dest = os.path.join(TARGET_DIR, filename)
|
||||
|
||||
if os.path.exists(dest):
|
||||
size = os.path.getsize(dest)
|
||||
print(f" [skip] {filename} already exists ({size / 1024 / 1024:.1f} MB)")
|
||||
return
|
||||
|
||||
print(f" [downloading] {filename} ...")
|
||||
try:
|
||||
urllib.request.urlretrieve(url, dest, reporthook=_progress)
|
||||
size = os.path.getsize(dest)
|
||||
print(f"\n [done] {filename} ({size / 1024 / 1024:.1f} MB)")
|
||||
except Exception as e:
|
||||
print(f"\n [error] {filename}: {e}")
|
||||
if os.path.exists(dest):
|
||||
os.remove(dest)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def _progress(block_num, block_size, total_size):
|
||||
downloaded = block_num * block_size
|
||||
if total_size > 0:
|
||||
pct = min(100, downloaded * 100 / total_size)
|
||||
mb = downloaded / 1024 / 1024
|
||||
total_mb = total_size / 1024 / 1024
|
||||
sys.stdout.write(f"\r {mb:.1f}/{total_mb:.1f} MB ({pct:.1f}%)")
|
||||
else:
|
||||
mb = downloaded / 1024 / 1024
|
||||
sys.stdout.write(f"\r {mb:.1f} MB downloaded")
|
||||
sys.stdout.flush()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print(f"Downloading LongMemEval cleaned-S dataset to: {TARGET_DIR}\n")
|
||||
for fname in FILES:
|
||||
download_file(fname)
|
||||
print("\nAll files downloaded successfully!")
|
||||
76
benchmark/kill.sh
Normal file
76
benchmark/kill.sh
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
#!/bin/bash
|
||||
# 杀死指定进程及其所有子进程
|
||||
# Usage: bash kill.sh <PID>
|
||||
|
||||
if [ -z "$1" ]; then
|
||||
echo "Usage: bash kill.sh <PID>"
|
||||
echo " 杀死指定进程及其所有子进程"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
PID=$1
|
||||
|
||||
# 检查进程是否存在
|
||||
if ! kill -0 "$PID" 2>/dev/null; then
|
||||
echo "进程 $PID 不存在"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 递归收集所有子进程（包括子进程的子进程）
|
||||
collect_children() {
|
||||
local parent=$1
|
||||
local children
|
||||
children=$(ps -o pid= --ppid "$parent" 2>/dev/null | tr -d ' ')
|
||||
for child in $children; do
|
||||
collect_children "$child"
|
||||
done
|
||||
echo "$parent"
|
||||
}
|
||||
|
||||
# 收集进程树（子进程在前，父进程在后，保证先杀子再杀父）
|
||||
PROCESS_TREE=$(collect_children "$PID")
|
||||
TOTAL=$(echo "$PROCESS_TREE" | wc -l | tr -d ' ')
|
||||
|
||||
echo "进程树（共 $TOTAL 个进程）:"
|
||||
while read -r p; do
|
||||
cmd=$(ps -o args= -p "$p" 2>/dev/null | head -c 80)
|
||||
printf " PID=%-8s %s\n" "$p" "$cmd"
|
||||
done <<< "$PROCESS_TREE"
|
||||
|
||||
# 先 SIGTERM 优雅终止
|
||||
echo ""
|
||||
echo "发送 SIGTERM..."
|
||||
while read -r p; do
|
||||
kill "$p" 2>/dev/null
|
||||
done <<< "$PROCESS_TREE"
|
||||
|
||||
# 等待最多 5 秒
|
||||
for i in $(seq 1 5); do
|
||||
alive=false
|
||||
while read -r p; do
|
||||
if kill -0 "$p" 2>/dev/null; then
|
||||
alive=true
|
||||
fi
|
||||
done <<< "$PROCESS_TREE"
|
||||
if [ "$alive" = false ]; then
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
# 检查是否还有残留，强制 SIGKILL
|
||||
remaining=false
|
||||
while read -r p; do
|
||||
if kill -0 "$p" 2>/dev/null; then
|
||||
remaining=true
|
||||
fi
|
||||
done <<< "$PROCESS_TREE"
|
||||
|
||||
if [ "$remaining" = true ]; then
|
||||
echo "部分进程未响应，发送 SIGKILL..."
|
||||
while read -r p; do
|
||||
kill -9 "$p" 2>/dev/null
|
||||
done <<< "$PROCESS_TREE"
|
||||
fi
|
||||
|
||||
echo "已终止进程树（根 PID=$PID，共 $TOTAL 个进程）"
|
||||
|
|
@ -1,141 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Remove generated LongMemEval files while keeping source inputs.
|
||||
|
||||
For each ``datasets/longmemeval/<idx>`` workspace, this keeps only:
|
||||
- query.json
|
||||
- answer.json
|
||||
- session/
|
||||
|
||||
All other files or directories in the sample root are considered generated
|
||||
artifacts and can be removed. AppleDouble files whose names start with ``._``
|
||||
are also removed recursively, including under ``session/``. The script is
|
||||
dry-run by default; pass ``--apply`` to actually delete. To delete only specific
|
||||
root-level generated files, pass one or more ``--filename`` values.
|
||||
|
||||
Examples:
|
||||
python benchmark/longmemeval/clean_sample_outputs.py
|
||||
python benchmark/longmemeval/clean_sample_outputs.py --apply
|
||||
python benchmark/longmemeval/clean_sample_outputs.py --start 36 --end 79 --apply
|
||||
python benchmark/longmemeval/clean_sample_outputs.py --filename check_golden.json --apply
|
||||
python benchmark/longmemeval/clean_sample_outputs.py --filename session_review.json --apply
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import shutil
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
DATA = REPO / "datasets" / "longmemeval"
|
||||
KEEP = {"query.json", "answer.json", "session"}
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
"""Parse command-line arguments."""
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--start", type=int, default=0, help="first numeric sample id to clean, inclusive (default 0)")
|
||||
p.add_argument("--end", type=int, default=499, help="last numeric sample id to clean, inclusive (default 499)")
|
||||
p.add_argument("--limit", type=int, default=0, help="only clean the first N selected samples (0 = all)")
|
||||
p.add_argument("--progress-every", type=int, default=25, help="print progress every N samples when applying")
|
||||
p.add_argument(
|
||||
"--filename",
|
||||
action="append",
|
||||
default=[],
|
||||
help="delete only this root-level file or directory name; can be passed multiple times",
|
||||
)
|
||||
p.add_argument("--apply", action="store_true", help="actually delete files; default is dry-run")
|
||||
return p.parse_args()
|
||||
|
||||
|
||||
def sample_ids() -> list[str]:
|
||||
"""List all numeric sample IDs."""
|
||||
ids = [p.name for p in DATA.iterdir() if p.is_dir() and p.name.isdigit()]
|
||||
return sorted(ids, key=int)
|
||||
|
||||
|
||||
def delete_path(path: Path) -> None:
|
||||
"""Delete a file, symlink, or directory."""
|
||||
if path.is_dir() and not path.is_symlink():
|
||||
shutil.rmtree(path)
|
||||
else:
|
||||
path.unlink()
|
||||
|
||||
|
||||
def iter_sample_targets(sample_dir: Path, filenames: set[str] | None = None) -> Iterator[Path]:
|
||||
"""Yield generated artifacts for one sample.
|
||||
|
||||
Root-level generated directories are yielded as a whole, so there is no
|
||||
need to recurse into them. AppleDouble files are only searched inside the
|
||||
kept ``session/`` directory.
|
||||
"""
|
||||
if filenames:
|
||||
for name in sorted(filenames):
|
||||
path = sample_dir / name
|
||||
if path.exists():
|
||||
yield path
|
||||
return
|
||||
|
||||
for path in sorted(sample_dir.iterdir(), key=lambda p: p.name):
|
||||
if path.name not in KEEP:
|
||||
yield path
|
||||
|
||||
session_dir = sample_dir / "session"
|
||||
if session_dir.is_dir():
|
||||
yield from session_dir.rglob("._*")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Main entry point."""
|
||||
args = parse_args()
|
||||
if args.end < args.start:
|
||||
raise ValueError(f"--end ({args.end}) must be >= --start ({args.start})")
|
||||
filenames = {name.strip() for name in args.filename if name.strip()}
|
||||
invalid_filenames = [name for name in filenames if Path(name).name != name]
|
||||
if invalid_filenames:
|
||||
raise ValueError(f"--filename only accepts root-level names, got: {invalid_filenames}")
|
||||
|
||||
ids = [idx for idx in sample_ids() if args.start <= int(idx) <= args.end]
|
||||
if args.limit:
|
||||
ids = ids[: args.limit]
|
||||
|
||||
total_targets = 0
|
||||
deleted = 0
|
||||
started_at = time.time()
|
||||
for ordinal, idx in enumerate(ids, start=1):
|
||||
sample_dir = DATA / idx
|
||||
sample_started_at = time.time()
|
||||
targets = list(iter_sample_targets(sample_dir, filenames=filenames))
|
||||
total_targets += len(targets)
|
||||
print(f"[sample {ordinal}/{len(ids)}] {idx} targets={len(targets)}", flush=True)
|
||||
for path in targets:
|
||||
if args.apply:
|
||||
target_started_at = time.time()
|
||||
print(f"[delete] {path}", flush=True)
|
||||
delete_path(path)
|
||||
deleted += 1
|
||||
print(f"[deleted] {path} elapsed={time.time() - target_started_at:.1f}s", flush=True)
|
||||
else:
|
||||
print(f"[would-delete] {path}")
|
||||
if args.apply and args.progress_every > 0 and (int(idx) + 1) % args.progress_every == 0:
|
||||
elapsed = time.time() - started_at
|
||||
print(
|
||||
f"[progress] processed={ordinal}/{len(ids)} through={idx} " f"deleted={deleted} elapsed={elapsed:.1f}s",
|
||||
flush=True,
|
||||
)
|
||||
print(f"[sample-done] {idx} elapsed={time.time() - sample_started_at:.1f}s", flush=True)
|
||||
|
||||
mode = "DELETE" if args.apply else "DRY-RUN"
|
||||
print(
|
||||
f"{mode} LongMemEval generated artifacts: samples={len(ids)} "
|
||||
f"targets={total_targets} deleted={deleted if args.apply else 0} range={args.start}..{args.end}",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
if not args.apply:
|
||||
print("No files deleted. Re-run with --apply to delete these paths.", flush=True)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
32
benchmark/longmemeval/config.yaml
Normal file
32
benchmark/longmemeval/config.yaml
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
# LongMemEval evaluation configuration
|
||||
# This file controls what/how to evaluate.
|
||||
|
||||
dataset:
|
||||
path: "benchmark/datasets/longmemeval/longmemeval_s_reme_cleaned.json"
|
||||
start_index: 0 # first item index
|
||||
num_items: 500 # how many items to evaluate (starting from start_index)
|
||||
max_sessions: 0 # 0 = all sessions; >0 = limit sessions per item for testing
|
||||
question_types: [] # filter by question_type; empty list = no filtering (all types)
|
||||
workspace_root: "benchmark/memory_workspaces/longmemeval-s" # workspace root for item workspaces
|
||||
|
||||
evaluation:
|
||||
# LLM-as-judge uses the 'judge' as_llm component defined in lme.yaml
|
||||
# Model and credentials are configured there (reading from .env)
|
||||
# Judgment is always binary (yes/no) — defined in lme/llm_judge.yaml
|
||||
num_workers: 32 # 0 = auto (cpu_count - 2, min 1); 1 = sequential; >1 = parallel
|
||||
filter_future_sessions: true # true = only ingest sessions with timestamp <= question_date
|
||||
|
||||
reme:
|
||||
config: "lme.yaml" # reme config to use (in reme/config/)
|
||||
# Dream trigger: when gap between consecutive sessions crosses this hour (23:00)
|
||||
dream_trigger_hour: 23
|
||||
# Dream scan_days for each trigger
|
||||
dream_scan_days: 2
|
||||
dream_max_units: 5
|
||||
|
||||
output:
|
||||
dir: "benchmark/results/longmemeval"
|
||||
log_dir: "logs" # log directory (relative to project root)
|
||||
log_prefix: "longmemeval" # benchmark name used in log filenames
|
||||
log_to_console: true
|
||||
log_to_file: true
|
||||
763
benchmark/longmemeval/run.py
Normal file
763
benchmark/longmemeval/run.py
Normal file
|
|
@ -0,0 +1,763 @@
|
|||
"""LongMemEval evaluation runner for ReMe.
|
||||
|
||||
Evaluates ReMe's long-term memory capability using the LongMemEval dataset.
|
||||
Each item gets an isolated workspace; sessions are ingested in chronological order;
|
||||
dream is triggered when sessions cross midnight (23:00); finally questions are
|
||||
answered via an agentic (ReAct) approach and judged by an LLM.
|
||||
|
||||
Usage:
|
||||
python benchmark/longmemeval/run.py
|
||||
python benchmark/longmemeval/run.py --config benchmark/longmemeval/config.yaml
|
||||
python benchmark/longmemeval/run.py -q # quiet: only eval-level logs
|
||||
python benchmark/longmemeval/run.py --log-level WARNING # reduce eval runner logs
|
||||
python benchmark/longmemeval/run.py --reme-log-level WARNING # reduce reme internal logs
|
||||
python benchmark/longmemeval/run.py --eval_only # query+judge only, reuse existing workspace
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import time
|
||||
import threading
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# Load .env from project root
|
||||
_PROJECT_ROOT = Path(__file__).parent.parent.parent
|
||||
load_dotenv(_PROJECT_ROOT / ".env")
|
||||
|
||||
# Workspace root for evaluation items — read from config.yaml (dataset.workspace_root)
|
||||
_WORKSPACE_ROOT_DEFAULT = "benchmark/memory_workspaces/longmemeval-s"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Logging
|
||||
# ---------------------------------------------------------------------------
|
||||
_DEFAULT_LOG_FORMAT = "%(asctime)s | %(levelname)s | %(message)s"
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format=_DEFAULT_LOG_FORMAT)
|
||||
logger = logging.getLogger("longmemeval")
|
||||
|
||||
# Noisy library loggers silenced by default
|
||||
_NOISY_LOGGERS = [
|
||||
"httpx",
|
||||
"httpcore",
|
||||
"openai",
|
||||
"uvicorn",
|
||||
"multipart",
|
||||
"asyncio",
|
||||
"watchfiles",
|
||||
"filelock",
|
||||
]
|
||||
|
||||
|
||||
def setup_logging(
|
||||
log_level: str,
|
||||
reme_log_level: str,
|
||||
log_dir: str | None = None,
|
||||
):
|
||||
"""Configure logging for the eval runner and reme internals.
|
||||
|
||||
Args:
|
||||
log_level: Level for the eval runner logger (DEBUG/INFO/WARNING/ERROR).
|
||||
reme_log_level: Level for reme's internal loguru logger.
|
||||
log_dir: Per-run log directory (absolute path). None = no file logging.
|
||||
"""
|
||||
numeric = getattr(logging, log_level.upper(), logging.INFO)
|
||||
# Eval runner logger
|
||||
logging.getLogger().setLevel(numeric)
|
||||
logger.setLevel(numeric)
|
||||
|
||||
# Suppress noisy library loggers when above DEBUG
|
||||
if numeric > logging.DEBUG:
|
||||
for name in _NOISY_LOGGERS:
|
||||
lib_logger = logging.getLogger(name)
|
||||
lib_logger.setLevel(max(numeric, logging.WARNING))
|
||||
|
||||
# Add file handler for eval runner if log_dir is specified
|
||||
if log_dir:
|
||||
os.makedirs(log_dir, exist_ok=True)
|
||||
log_filepath = os.path.join(log_dir, "runner.log")
|
||||
file_handler = logging.FileHandler(log_filepath, encoding="utf-8")
|
||||
file_handler.setLevel(numeric)
|
||||
file_handler.setFormatter(logging.Formatter(_DEFAULT_LOG_FORMAT))
|
||||
logging.getLogger().addHandler(file_handler)
|
||||
logger.info(f"Eval runner log file: {log_filepath}")
|
||||
|
||||
# Reme internal logger (loguru) — will be applied per-worker via _configure_worker
|
||||
os.environ["REME_LOG_LEVEL"] = reme_log_level.upper()
|
||||
if log_dir:
|
||||
os.environ["REME_LOG_DIR"] = log_dir
|
||||
|
||||
|
||||
def _configure_worker(
|
||||
log_level: str,
|
||||
reme_log_level: str,
|
||||
log_dir: str | None = None,
|
||||
):
|
||||
"""Set up logging inside a multiprocessing worker process.
|
||||
|
||||
Must be called at the top of each worker because child processes inherit
|
||||
parent state but loguru sinks are NOT shared across fork/spawn.
|
||||
"""
|
||||
numeric = getattr(logging, log_level.upper(), logging.INFO)
|
||||
logging.basicConfig(level=numeric, format=_DEFAULT_LOG_FORMAT, force=True)
|
||||
logging.getLogger("longmemeval").setLevel(numeric)
|
||||
if numeric > logging.DEBUG:
|
||||
for name in _NOISY_LOGGERS:
|
||||
logging.getLogger(name).setLevel(max(numeric, logging.WARNING))
|
||||
|
||||
# Add file handler for eval runner in worker process
|
||||
if log_dir:
|
||||
os.makedirs(log_dir, exist_ok=True)
|
||||
pid = os.getpid()
|
||||
log_filepath = os.path.join(log_dir, f"worker-{pid}.log")
|
||||
file_handler = logging.FileHandler(log_filepath, encoding="utf-8")
|
||||
file_handler.setLevel(numeric)
|
||||
file_handler.setFormatter(logging.Formatter(_DEFAULT_LOG_FORMAT))
|
||||
logging.getLogger().addHandler(file_handler)
|
||||
|
||||
# Re-initialize loguru for reme internals at the desired level
|
||||
from reme.utils import get_logger
|
||||
|
||||
reme_log_dir = log_dir or "logs"
|
||||
get_logger(log_dir=reme_log_dir, level=reme_log_level.upper(), force_init=True)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Config loading
|
||||
# ---------------------------------------------------------------------------
|
||||
def load_eval_config(config_path: str | None = None) -> dict:
|
||||
"""Load evaluation config yaml with env-var expansion."""
|
||||
if config_path is None:
|
||||
config_path = str(Path(__file__).parent / "config.yaml")
|
||||
with open(config_path, encoding="utf-8") as f:
|
||||
raw = f.read()
|
||||
|
||||
# Expand ${VAR} and ${VAR:-default}
|
||||
def _expand(m):
|
||||
expr = m.group(1)
|
||||
if ":-" in expr:
|
||||
key, default = expr.split(":-", 1)
|
||||
return os.environ.get(key, default)
|
||||
return os.environ.get(expr, "")
|
||||
|
||||
raw = re.sub(r"\$\{([^}]+)\}", _expand, raw)
|
||||
return yaml.safe_load(raw)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Date utilities
|
||||
# ---------------------------------------------------------------------------
|
||||
def parse_haystack_date(date_str: str) -> datetime:
|
||||
"""Parse LongMemEval date format: '2023/05/20 (Sat) 02:21' -> datetime."""
|
||||
m = re.match(r"(\d{4}/\d{2}/\d{2})\s+\(\w+\)\s+(\d{2}:\d{2})", date_str)
|
||||
if not m:
|
||||
raise ValueError(f"Cannot parse haystack date: {date_str!r}")
|
||||
return datetime.strptime(f"{m.group(1)} {m.group(2)}", "%Y/%m/%d %H:%M")
|
||||
|
||||
|
||||
def to_iso(dt: datetime) -> str:
|
||||
"""Convert datetime to ISO-8601 string precise to seconds."""
|
||||
return dt.strftime("%Y-%m-%dT%H:%M:%S")
|
||||
|
||||
|
||||
def should_trigger_dream(prev_dt: datetime, curr_dt: datetime, _trigger_hour: int = 23) -> bool:
|
||||
"""Check if the time gap between two sessions crosses trigger_hour (e.g. 23:00)."""
|
||||
if prev_dt.date() == curr_dt.date():
|
||||
return False
|
||||
# There's at least one midnight crossing; check if trigger_hour is between them
|
||||
# Simple heuristic: if dates differ, dream should run for the previous day
|
||||
return True
|
||||
|
||||
|
||||
def sessions_sorted_by_time(item: dict) -> list[tuple[int, datetime, str, list[dict]]]:
|
||||
"""Return (original_index, parsed_datetime, session_id, messages) sorted by time."""
|
||||
entries = []
|
||||
for i, (date_str, sid, msgs) in enumerate(
|
||||
zip(item["haystack_dates"], item["haystack_session_ids"], item["haystack_sessions"]),
|
||||
):
|
||||
dt = parse_haystack_date(date_str)
|
||||
entries.append((i, dt, sid, msgs))
|
||||
# Sort by time (ascending)
|
||||
entries.sort(key=lambda x: x[1])
|
||||
return entries
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Message formatting
|
||||
# ---------------------------------------------------------------------------
|
||||
def format_messages_for_reme(messages: list[dict], session_dt: datetime) -> list[dict]:
|
||||
"""Convert LongMemEval messages to ReMe auto_memory format.
|
||||
|
||||
Adds: name, created_at (ISO seconds). All messages in a session share the
|
||||
same created_at (the session timestamp).
|
||||
"""
|
||||
formatted = []
|
||||
for msg in messages:
|
||||
role = msg["role"]
|
||||
formatted.append(
|
||||
{
|
||||
"name": role,
|
||||
"role": role,
|
||||
"content": msg["content"],
|
||||
"created_at": to_iso(session_dt),
|
||||
},
|
||||
)
|
||||
return formatted
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# LLM-as-Judge (delegated to answer_judge_step via app.run_job)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def judge_response_via_job(
|
||||
app,
|
||||
question: str,
|
||||
ground_truth: str,
|
||||
response: str,
|
||||
question_type: str,
|
||||
) -> dict:
|
||||
"""Use the answer_judge_step to evaluate a response against the golden answer."""
|
||||
judge_resp = await app.run_job(
|
||||
"answer_judge",
|
||||
query=question,
|
||||
agent_answer=response,
|
||||
golden_answer=ground_truth,
|
||||
question_type=question_type,
|
||||
)
|
||||
|
||||
verdict = (judge_resp.answer or "").strip().lower()
|
||||
raw_answer = (judge_resp.metadata or {}).get("raw_answer_judgement", "")
|
||||
|
||||
return {
|
||||
"verdict": verdict,
|
||||
"reason": raw_answer if verdict not in ("yes", "no") else "",
|
||||
"metric": "binary",
|
||||
"question_type": question_type,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main evaluation pipeline
|
||||
# ---------------------------------------------------------------------------
|
||||
async def evaluate_item(item: dict, eval_config: dict, item_index: int, eval_only: bool = False) -> dict:
|
||||
"""Evaluate a single LongMemEval item end-to-end.
|
||||
|
||||
Args:
|
||||
item: The dataset item containing question, answer, sessions, etc.
|
||||
eval_config: The evaluation configuration dict.
|
||||
item_index: The index of this item in the dataset.
|
||||
eval_only: If True, skip ingestion (phases 1-3) and only run query+judge
|
||||
using the existing workspace. Useful for re-evaluating different query
|
||||
configurations without re-ingesting sessions.
|
||||
"""
|
||||
from reme import Application
|
||||
from reme.config import resolve_app_config
|
||||
|
||||
reme_cfg = eval_config["reme"]
|
||||
dream_trigger_hour = reme_cfg.get("dream_trigger_hour", 23)
|
||||
dream_scan_days = reme_cfg.get("dream_scan_days", 2)
|
||||
dream_max_units = reme_cfg.get("dream_max_units", 5)
|
||||
|
||||
# Sort sessions by time
|
||||
sorted_sessions = sessions_sorted_by_time(item)
|
||||
|
||||
# Filter out sessions that occur after question_date (if enabled)
|
||||
filter_future = eval_config["evaluation"].get("filter_future_sessions", True)
|
||||
if filter_future and item.get("question_date"):
|
||||
question_dt = parse_haystack_date(item["question_date"])
|
||||
total_before_filter = len(sorted_sessions)
|
||||
sorted_sessions = [(i, dt, sid, msgs) for i, dt, sid, msgs in sorted_sessions if dt <= question_dt]
|
||||
if len(sorted_sessions) < total_before_filter:
|
||||
logger.info(
|
||||
f"[Item {item_index}] Filtered sessions: {total_before_filter} -> {len(sorted_sessions)} "
|
||||
f"(removed {total_before_filter - len(sorted_sessions)} future sessions "
|
||||
f"after question_date={item['question_date']})",
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"[Item %s] question_id=%s type=%s sessions=%d%s",
|
||||
item_index,
|
||||
item["question_id"],
|
||||
item["question_type"],
|
||||
len(sorted_sessions),
|
||||
" [eval_only]" if eval_only else "",
|
||||
)
|
||||
|
||||
# Use fixed workspace directory (clean it for fresh evaluation)
|
||||
workspace_root = _PROJECT_ROOT / eval_config["dataset"].get("workspace_root", _WORKSPACE_ROOT_DEFAULT)
|
||||
item_dir = workspace_root / f"item_{item_index}"
|
||||
workspace_dir = str(item_dir / ".reme")
|
||||
if eval_only:
|
||||
if not item_dir.exists() or not Path(workspace_dir).exists():
|
||||
raise FileNotFoundError(
|
||||
f"[Item {item_index}] eval_only: workspace not found at {item_dir}. "
|
||||
f"Run without --eval_only first to build the workspace.",
|
||||
)
|
||||
else:
|
||||
if item_dir.exists():
|
||||
shutil.rmtree(item_dir)
|
||||
logger.info(f"[Item {item_index}] Cleaned existing workspace: {item_dir}")
|
||||
else:
|
||||
logger.info(f"[Item {item_index}] Workspace not found, creating: {item_dir}")
|
||||
item_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Pre-initialize ReMe's loguru logger with the correct log_dir
|
||||
# (singleton — Application.__init__ will reuse this instance)
|
||||
output_cfg = eval_config.get("output", {})
|
||||
if output_cfg.get("log_to_file", False):
|
||||
reme_log_dir = os.environ.get("REME_LOG_DIR")
|
||||
if reme_log_dir:
|
||||
from reme.utils import get_logger
|
||||
|
||||
get_logger(
|
||||
log_dir=reme_log_dir,
|
||||
level=os.environ.get("REME_LOG_LEVEL", "INFO"),
|
||||
log_to_console=output_cfg.get("log_to_console", True),
|
||||
log_to_file=True,
|
||||
force_init=True,
|
||||
)
|
||||
|
||||
cfg = resolve_app_config(
|
||||
config=reme_cfg["config"],
|
||||
workspace_dir=workspace_dir,
|
||||
log_to_console=output_cfg.get("log_to_console", True),
|
||||
log_to_file=output_cfg.get("log_to_file", False),
|
||||
enable_logo=False,
|
||||
)
|
||||
|
||||
app = Application(**cfg)
|
||||
await app.start()
|
||||
|
||||
try:
|
||||
dream_dates_triggered = set()
|
||||
dream_available = True # Set to False if auto_dream job is not found
|
||||
|
||||
if not eval_only:
|
||||
# ── Phase 1: Ingest sessions ──────────────────────────────
|
||||
prev_dt = None
|
||||
|
||||
for idx, (_, session_dt, session_id, messages) in enumerate(sorted_sessions):
|
||||
# Check if dream should be triggered before this session
|
||||
if (
|
||||
dream_available
|
||||
and prev_dt is not None
|
||||
and should_trigger_dream(prev_dt, session_dt, dream_trigger_hour)
|
||||
):
|
||||
dream_date = prev_dt.strftime("%Y-%m-%d")
|
||||
if dream_date not in dream_dates_triggered:
|
||||
logger.info(f"[Item {item_index}] Triggering dream for date={dream_date}")
|
||||
try:
|
||||
dream_resp = await app.run_job(
|
||||
"auto_dream",
|
||||
date=dream_date,
|
||||
scan_days=dream_scan_days,
|
||||
max_units=dream_max_units,
|
||||
)
|
||||
logger.info(
|
||||
f"[Item {item_index}] Dream done: success={dream_resp.success} "
|
||||
f"answer={dream_resp.answer[:100] if dream_resp.answer else ''}",
|
||||
)
|
||||
except Exception as e:
|
||||
if "not found" in str(e).lower():
|
||||
dream_available = False
|
||||
logger.warning(f"[Item {item_index}] auto_dream job not found, skipping all dreams")
|
||||
else:
|
||||
logger.warning(f"[Item {item_index}] Dream failed for {dream_date}: {e}")
|
||||
dream_dates_triggered.add(dream_date)
|
||||
# Index update after dream to pick up new digest nodes
|
||||
await app.run_job("index_update")
|
||||
|
||||
# Format and ingest the session
|
||||
formatted_msgs = format_messages_for_reme(messages, session_dt)
|
||||
date_str = session_dt.strftime("%Y-%m-%d")
|
||||
|
||||
logger.info(
|
||||
f"[Item {item_index}] Ingesting session {idx+1}/{len(sorted_sessions)} "
|
||||
f"id={session_id} date={date_str} msgs={len(formatted_msgs)}",
|
||||
)
|
||||
resp = await app.run_job(
|
||||
"auto_memory",
|
||||
messages=formatted_msgs,
|
||||
session_id=session_id,
|
||||
date=date_str,
|
||||
)
|
||||
if not resp.success:
|
||||
logger.warning(
|
||||
f"[Item {item_index}] auto_memory failed for session {session_id}: {resp.answer}",
|
||||
)
|
||||
|
||||
# Manual index update after each session
|
||||
await app.run_job("index_update")
|
||||
|
||||
prev_dt = session_dt
|
||||
|
||||
# ── Phase 2: Final dream for the last day ─────────────────
|
||||
if dream_available and prev_dt is not None:
|
||||
last_dream_date = prev_dt.strftime("%Y-%m-%d")
|
||||
if last_dream_date not in dream_dates_triggered:
|
||||
logger.info(f"[Item {item_index}] Final dream for date={last_dream_date}")
|
||||
try:
|
||||
await app.run_job(
|
||||
"auto_dream",
|
||||
date=last_dream_date,
|
||||
scan_days=dream_scan_days,
|
||||
max_units=dream_max_units,
|
||||
)
|
||||
except Exception as e:
|
||||
if "not found" in str(e).lower():
|
||||
dream_available = False
|
||||
logger.warning(f"[Item {item_index}] auto_dream job not found, skipping all dreams")
|
||||
else:
|
||||
logger.warning(f"[Item {item_index}] Final dream failed: {e}")
|
||||
dream_dates_triggered.add(last_dream_date)
|
||||
# Index update after final dream
|
||||
await app.run_job("index_update")
|
||||
|
||||
# ── Phase 3: Digest update ────────────────────────────────
|
||||
await app.run_job("digest_update")
|
||||
|
||||
# ── Phase 4: Ask question via agentic_answer job (ReAct agent) ──
|
||||
question = item["question"]
|
||||
question_date_raw = item.get("question_date", "")
|
||||
question_dt = parse_haystack_date(question_date_raw) if question_date_raw else None
|
||||
query_time = to_iso(question_dt) if question_dt else ""
|
||||
logger.info(
|
||||
f"[Item {item_index}] Asking (agentic): {question[:80]}... query_time={query_time}",
|
||||
)
|
||||
|
||||
query_resp = await app.run_job(
|
||||
"agentic_answer",
|
||||
query=question,
|
||||
query_time=query_time,
|
||||
)
|
||||
agentic_response = (query_resp.answer or "").strip()
|
||||
if not agentic_response:
|
||||
agentic_response = "(no answer generated)"
|
||||
|
||||
logger.info(f"[Item {item_index}] Agentic response: {agentic_response[:200]}...")
|
||||
|
||||
# ── Phase 5: Judge agentic response (via answer_judge_step) ──────────
|
||||
logger.info(f"[Item {item_index}] Judging agentic (binary, type={item['question_type']})...")
|
||||
agentic_judgment = await judge_response_via_job(
|
||||
app=app,
|
||||
question=question,
|
||||
ground_truth=item["answer"],
|
||||
response=agentic_response,
|
||||
question_type=item["question_type"],
|
||||
)
|
||||
logger.info(f"[Item {item_index}] agentic binary result: {agentic_judgment}")
|
||||
|
||||
finally:
|
||||
await app.close()
|
||||
|
||||
return {
|
||||
"question_id": item["question_id"],
|
||||
"question_type": item["question_type"],
|
||||
"question": question,
|
||||
"ground_truth": item["answer"],
|
||||
"agentic_response": agentic_response,
|
||||
"agentic_judgment": agentic_judgment,
|
||||
"sessions_ingested": len(sorted_sessions),
|
||||
"dreams_triggered": len(dream_dates_triggered),
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Worker: runs a single item in its own process with its own event loop
|
||||
# ---------------------------------------------------------------------------
|
||||
def _evaluate_item_worker(task_input: tuple) -> dict:
|
||||
"""Worker function for multiprocessing. Each process gets its own event loop."""
|
||||
item, eval_config, item_index, log_level, reme_log_level, eval_only, log_dir = task_input
|
||||
import asyncio # pylint: disable=import-outside-toplevel
|
||||
|
||||
_configure_worker(log_level, reme_log_level, log_dir=log_dir)
|
||||
|
||||
# Permanently suppress "Task exception was never retrieved" /
|
||||
# "Event loop is closed" noise from httpx AsyncClient GC cleanup.
|
||||
# These fire AFTER asyncio.run() closes the loop, during Python's
|
||||
# garbage collection of httpx connection-pool tasks — harmless.
|
||||
logging.getLogger("asyncio").setLevel(logging.CRITICAL)
|
||||
|
||||
return asyncio.run(evaluate_item(item, eval_config, item_index, eval_only=eval_only))
|
||||
|
||||
|
||||
def _indexed_worker(indexed_input: tuple) -> tuple:
|
||||
"""Module-level wrapper for imap_unordered with index tracking."""
|
||||
idx, task_input = indexed_input
|
||||
return idx, _evaluate_item_worker(task_input)
|
||||
|
||||
|
||||
def _resolve_num_workers(configured: int) -> int:
|
||||
"""Resolve num_workers: 0=auto (cpu_count-2, min 1), 1=sequential, >1=parallel."""
|
||||
if configured == 0:
|
||||
return max(1, (os.cpu_count() or 4) - 2)
|
||||
return max(1, configured)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Entry point
|
||||
# ---------------------------------------------------------------------------
|
||||
def main(
|
||||
config_path: str | None = None,
|
||||
log_level: str = "INFO",
|
||||
reme_log_level: str = "INFO",
|
||||
eval_only: bool = False,
|
||||
):
|
||||
"""Run the LongMemEval evaluation pipeline.
|
||||
|
||||
Args:
|
||||
config_path: Path to the YAML config file.
|
||||
log_level: Log level for the eval runner.
|
||||
reme_log_level: Log level for reme internal logs.
|
||||
eval_only: If True, skip ingestion and only run query+judge using
|
||||
existing workspaces.
|
||||
"""
|
||||
from multiprocessing import Pool # pylint: disable=import-outside-toplevel
|
||||
|
||||
# Load config BEFORE logging setup so log_dir is available
|
||||
eval_config = load_eval_config(config_path)
|
||||
|
||||
# Resolve per-run log directory from config
|
||||
output_cfg = eval_config.get("output", {})
|
||||
log_dir_abs = None
|
||||
if output_cfg.get("log_to_file", False):
|
||||
log_dir_raw = output_cfg.get("log_dir", "logs")
|
||||
log_prefix = output_cfg.get("log_prefix", "longmemeval")
|
||||
run_ts = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
|
||||
log_dir_abs = str(_PROJECT_ROOT / log_dir_raw / f"{log_prefix}_{run_ts}")
|
||||
|
||||
setup_logging(log_level, reme_log_level, log_dir=log_dir_abs)
|
||||
dataset_cfg = eval_config["dataset"]
|
||||
|
||||
# Load dataset
|
||||
dataset_path = _PROJECT_ROOT / dataset_cfg["path"]
|
||||
logger.info(f"Loading dataset from {dataset_path}")
|
||||
with open(dataset_path, encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
start = dataset_cfg.get("start_index", 0)
|
||||
num_items = dataset_cfg.get("num_items", 0)
|
||||
if num_items > 0:
|
||||
raw_items = data[start : start + num_items]
|
||||
else:
|
||||
raw_items = data[start:]
|
||||
|
||||
# Build item list
|
||||
items_with_idx = [(start + i, item) for i, item in enumerate(raw_items)]
|
||||
|
||||
# Filter by question_type if specified
|
||||
question_types = dataset_cfg.get("question_types") or []
|
||||
if question_types:
|
||||
before_filter = len(items_with_idx)
|
||||
items_with_idx = [(idx, item) for idx, item in items_with_idx if item.get("question_type") in question_types]
|
||||
logger.info(
|
||||
f"Filtered by question_types={question_types}: {before_filter} -> {len(items_with_idx)} items",
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Evaluating %d item(s) starting from index %d%s",
|
||||
len(items_with_idx),
|
||||
start,
|
||||
" [eval_only: query+judge only]" if eval_only else "",
|
||||
)
|
||||
|
||||
# Resolve parallelism
|
||||
num_workers = _resolve_num_workers(eval_config["evaluation"].get("num_workers", 1))
|
||||
logger.info(f"Using {num_workers} worker(s)")
|
||||
|
||||
# Create output directory
|
||||
output_dir = _PROJECT_ROOT / output_cfg.get("dir", "benchmark/results/longmemeval")
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Create workspace root directory
|
||||
workspace_root = _PROJECT_ROOT / dataset_cfg.get("workspace_root", _WORKSPACE_ROOT_DEFAULT)
|
||||
workspace_root.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Pre-check: verify all workspaces exist in eval_only mode
|
||||
if eval_only:
|
||||
missing_items = []
|
||||
for orig_idx, _ in items_with_idx:
|
||||
item_dir = workspace_root / f"item_{orig_idx}"
|
||||
if not item_dir.exists() or not (item_dir / ".reme").exists():
|
||||
missing_items.append(orig_idx)
|
||||
if missing_items:
|
||||
preview = missing_items[:10]
|
||||
suffix = "..." if len(missing_items) > 10 else ""
|
||||
raise FileNotFoundError(
|
||||
f"eval_only: {len(missing_items)} workspace(s) not found under {workspace_root}. "
|
||||
f"Missing item indices: {preview}{suffix}. "
|
||||
f"Run without --eval_only first to build the workspaces.",
|
||||
)
|
||||
|
||||
# Build task args — include log levels, eval_only flag, and log paths (use original index for workspace lookup)
|
||||
task_args = [
|
||||
(item, eval_config, orig_idx, log_level, reme_log_level, eval_only, log_dir_abs)
|
||||
for orig_idx, item in items_with_idx
|
||||
]
|
||||
|
||||
# Progress tracking (force print regardless of log level, every 10 minutes)
|
||||
total_items = len(task_args)
|
||||
completed_count = [0] # use list for mutability in closure
|
||||
start_time = time.time()
|
||||
progress_lock = threading.Lock()
|
||||
|
||||
def _print_progress(prefix: str = "PROGRESS"):
|
||||
elapsed = time.time() - start_time
|
||||
elapsed_min = elapsed / 60
|
||||
done = completed_count[0]
|
||||
pct = 100.0 * done / total_items if total_items else 0
|
||||
eta_str = "N/A"
|
||||
if done > 0:
|
||||
eta_sec = elapsed / done * (total_items - done)
|
||||
eta_str = f"{eta_sec/60:.1f}min"
|
||||
print(
|
||||
f"[{prefix}] {datetime.now().strftime('%Y-%m-%d %H:%M:%S')} | "
|
||||
f"{done}/{total_items} ({pct:.1f}%) completed | "
|
||||
f"elapsed={elapsed_min:.1f}min | ETA={eta_str}",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
def _progress_timer():
|
||||
"""Background thread: print progress every 10 minutes."""
|
||||
while not _timer_stop.is_set():
|
||||
_timer_stop.wait(600) # 10 minutes
|
||||
if not _timer_stop.is_set():
|
||||
with progress_lock:
|
||||
_print_progress()
|
||||
|
||||
_timer_stop = threading.Event()
|
||||
timer_thread = threading.Thread(target=_progress_timer, daemon=True)
|
||||
timer_thread.start()
|
||||
|
||||
# Run evaluation
|
||||
if num_workers == 1:
|
||||
# Sequential mode
|
||||
results = []
|
||||
for task_input in task_args:
|
||||
result = _evaluate_item_worker(task_input)
|
||||
results.append(result)
|
||||
with progress_lock:
|
||||
completed_count[0] += 1
|
||||
else:
|
||||
# Parallel mode — use imap_unordered for progress tracking
|
||||
results = [None] * total_items
|
||||
indexed_args = list(enumerate(task_args))
|
||||
|
||||
with Pool(processes=num_workers) as pool:
|
||||
for idx, result in pool.imap_unordered(_indexed_worker, indexed_args):
|
||||
results[idx] = result
|
||||
with progress_lock:
|
||||
completed_count[0] += 1
|
||||
|
||||
# Stop progress timer
|
||||
_timer_stop.set()
|
||||
timer_thread.join(timeout=2)
|
||||
|
||||
# Save results
|
||||
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
output_file = output_dir / f"results_{timestamp}.json"
|
||||
with open(output_file, "w", encoding="utf-8") as f:
|
||||
json.dump(results, f, ensure_ascii=False, indent=2)
|
||||
logger.info(f"Results saved to {output_file}")
|
||||
|
||||
# Final progress
|
||||
_print_progress("FINAL")
|
||||
|
||||
_print_summary(results, start_time)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Summary printing
|
||||
# ---------------------------------------------------------------------------
|
||||
def _print_summary(results: list[dict], start_time: float) -> None:
|
||||
"""Print per-item verdicts and per-type accuracy."""
|
||||
print("\n" + "=" * 60)
|
||||
print("EVALUATION RESULTS")
|
||||
print("=" * 60)
|
||||
|
||||
def _accumulate(judgment_key):
|
||||
correct = 0
|
||||
stats: dict = {} # {question_type: {correct: int, total: int}}
|
||||
for r in results:
|
||||
qtype = r["question_type"]
|
||||
verdict = r.get(judgment_key, {}).get("verdict", "N/A")
|
||||
if qtype not in stats:
|
||||
stats[qtype] = {"correct": 0, "total": 0}
|
||||
stats[qtype]["total"] += 1
|
||||
if verdict == "yes":
|
||||
correct += 1
|
||||
stats[qtype]["correct"] += 1
|
||||
return correct, stats
|
||||
|
||||
agentic_correct, agentic_type_stats = _accumulate("agentic_judgment")
|
||||
|
||||
total = len(results)
|
||||
|
||||
# Per-item verdict rows
|
||||
for r in results:
|
||||
a_verdict = r.get("agentic_judgment", {}).get("verdict", "N/A")
|
||||
print(f" [{r['question_id']}] type={r['question_type']} agentic={a_verdict}")
|
||||
|
||||
print("\n" + "-" * 60)
|
||||
print(f" Items: {total}")
|
||||
|
||||
# Agentic stats
|
||||
print("\n ── Agentic (ReAct) ──")
|
||||
print(f" Overall accuracy: {agentic_correct}/{total} ({100*agentic_correct/total:.1f}%)")
|
||||
print(" Per-type accuracy:")
|
||||
for qtype, stats in sorted(agentic_type_stats.items()):
|
||||
acc = 100 * stats["correct"] / stats["total"] if stats["total"] else 0
|
||||
print(f" {qtype}: {stats['correct']}/{stats['total']} ({acc:.1f}%)")
|
||||
|
||||
print("=" * 60)
|
||||
total_elapsed = time.time() - start_time
|
||||
print(f"\n Total time: {total_elapsed/60:.1f} min")
|
||||
print("\n" + "=" * 60)
|
||||
print(" [DONE] EVALUATION COMPLETED SUCCESSFULLY")
|
||||
print("=" * 60 + "\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(description="LongMemEval evaluation runner")
|
||||
parser.add_argument("--config", type=str, default=None, help="Path to config.yaml")
|
||||
parser.add_argument(
|
||||
"--log-level",
|
||||
type=str,
|
||||
default="INFO",
|
||||
choices=["DEBUG", "INFO", "WARNING", "ERROR"],
|
||||
help="Log level for the eval runner (default: INFO)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--reme-log-level",
|
||||
type=str,
|
||||
default="INFO",
|
||||
choices=["DEBUG", "INFO", "WARNING", "ERROR"],
|
||||
help="Log level for reme internal logs — loguru (default: INFO)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"-q",
|
||||
"--quiet",
|
||||
action="store_true",
|
||||
help="Shortcut for --log-level WARNING --reme-log-level WARNING",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--eval_only",
|
||||
action="store_true",
|
||||
help="Skip ingestion (phases 1-3). Reuse existing workspaces and only run query+judge.",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.quiet:
|
||||
args.log_level = "WARNING"
|
||||
args.reme_log_level = "WARNING"
|
||||
|
||||
main(args.config, args.log_level, args.reme_log_level, eval_only=args.eval_only)
|
||||
|
|
@ -1,343 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Drive the LongMemEval memory pipeline across all samples.
|
||||
|
||||
For every workspace under ``datasets/longmemeval/<idx>`` this launches one or more
|
||||
``reme start config=jinli_lme job=<job>`` runs with ``LME_WORKSPACE_DIR`` pointed
|
||||
at that sample. The pipeline jobs, in order, are:
|
||||
|
||||
1. auto_memory — distil every raw session into a daily note (``daily/*.md``)
|
||||
2. update_index — clear the store and rebuild the index over ``daily/*.md``
|
||||
3. agentic_answer — read ``query.json`` and answer it, writing ``mem_answer.json``
|
||||
4. llm_judge — judge ``mem_answer.json`` against ``answer.json``
|
||||
|
||||
Pick one with ``--job``, or ``--job all`` to run the full pipeline *serially per sample*.
|
||||
Runs are capped at ``--concurrency`` (default 1 for ``--job auto_memory``, otherwise
|
||||
3) samples at once and each launch is staggered by ``--stagger`` seconds so they
|
||||
do not all hit the LLM API at once.
|
||||
|
||||
By default every selected job is rerun for every sample — each job's own clear
|
||||
step (configured in jinli_lme.yaml) wipes stale output first, so a run is always
|
||||
a clean rebuild. Pass ``--resume`` to instead skip samples whose output already
|
||||
exists (``daily/`` for auto_memory, ``metadata/embedding_store/`` for
|
||||
update_index, ``mem_answer.json`` for agentic_answer, ``mem_answer.json`` with
|
||||
``llm_judge.judgement`` for llm_judge) and continue an interrupted batch. Each
|
||||
sample's stdout/stderr goes to ``logs/agentic_answer/<job>/<idx>.log``.
|
||||
|
||||
After an agentic_answer run finishes, the driver aggregates every sample's query,
|
||||
golden answer, predicted answer, LLM judgement and a best-effort tool-call trail
|
||||
into one big JSON at ``logs/agentic_answer/aggregate.json``.
|
||||
|
||||
Examples:
|
||||
python benchmark/longmemeval/run_agentic_answer.py # agentic_answer, all 500, conc 3
|
||||
python benchmark/longmemeval/run_agentic_answer.py --job all # full pipeline serially per sample
|
||||
python benchmark/longmemeval/run_agentic_answer.py --job auto_memory # just step 1
|
||||
python benchmark/longmemeval/run_agentic_answer.py --job llm_judge # just judge existing answers
|
||||
python benchmark/longmemeval/run_agentic_answer.py --limit 5 --dry-run # list what would run
|
||||
python benchmark/longmemeval/run_agentic_answer.py --start 187 # samples 187..499
|
||||
python benchmark/longmemeval/run_agentic_answer.py --start 187 --end 499 # samples 187..499
|
||||
python benchmark/longmemeval/run_agentic_answer.py --job all --resume # continue an interrupted batch
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
DATA = REPO / "datasets" / "longmemeval"
|
||||
LOGDIR = REPO / "logs" / "agentic_answer"
|
||||
AGGREGATE = LOGDIR / "aggregate.json"
|
||||
|
||||
# Pipeline jobs in execution order.
|
||||
JOB_ORDER = ["auto_memory", "update_index", "agentic_answer", "llm_judge"]
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
"""Parse command-line arguments."""
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument(
|
||||
"--job",
|
||||
choices=[*JOB_ORDER, "all"],
|
||||
default="agentic_answer",
|
||||
help="which job to run per sample; 'all' runs the full pipeline serially (default: agentic_answer)",
|
||||
)
|
||||
p.add_argument("--concurrency", type=int, default=1, help="max samples running at once (default 3)")
|
||||
p.add_argument("--stagger", type=float, default=1.0, help="seconds between consecutive launches (default 1)")
|
||||
p.add_argument("--start", type=int, default=0, help="first numeric sample id to process, inclusive (default 0)")
|
||||
p.add_argument(
|
||||
"--end",
|
||||
type=int,
|
||||
default=0,
|
||||
help="last numeric sample id to process, inclusive (0 = no upper bound)",
|
||||
)
|
||||
p.add_argument("--limit", type=int, default=0, help="only process the first N samples (0 = all)")
|
||||
p.add_argument(
|
||||
"--resume",
|
||||
action="store_true",
|
||||
help="skip a sample when the job's output already exists (resume an interrupted run); "
|
||||
"by default every selected job is rerun so the config's clear step rebuilds cleanly",
|
||||
)
|
||||
p.add_argument("--dry-run", action="store_true", help="list what would run, launch nothing")
|
||||
p.add_argument("--no-aggregate", action="store_true", help="skip writing aggregate.json after answer/judge jobs")
|
||||
return p.parse_args()
|
||||
|
||||
|
||||
def selected_jobs(job: str) -> list[str]:
|
||||
"""Expand the --job choice into an ordered list of jobs."""
|
||||
return list(JOB_ORDER) if job == "all" else [job]
|
||||
|
||||
|
||||
def sample_ids() -> list[str]:
|
||||
"""List all sample IDs (numeric workspace dirs), numerically sorted."""
|
||||
ids = [p.name for p in DATA.iterdir() if p.is_dir() and p.name.isdigit()]
|
||||
return sorted(ids, key=int)
|
||||
|
||||
|
||||
def job_done(idx: str, job: str) -> bool:
|
||||
"""Return True when ``job``'s expected output already exists for sample ``idx``."""
|
||||
ws = DATA / idx
|
||||
if job == "auto_memory":
|
||||
daily = ws / "daily"
|
||||
return daily.is_dir() and any(daily.rglob("*.md"))
|
||||
if job == "update_index":
|
||||
store = ws / "metadata" / "embedding_store"
|
||||
return store.is_dir() and any(store.iterdir())
|
||||
if job == "agentic_answer":
|
||||
return (ws / "mem_answer.json").exists()
|
||||
if job == "llm_judge":
|
||||
judge = _load_json(ws / "mem_answer.json").get("llm_judge")
|
||||
return isinstance(judge, dict) and bool(str(judge.get("judgement") or "").strip())
|
||||
raise ValueError(f"unknown job: {job}")
|
||||
|
||||
|
||||
async def run_job(idx: str, job: str, counters: dict) -> bool:
|
||||
"""Run a single job for a single sample. Returns True on success."""
|
||||
log = LOGDIR / job / f"{idx}.log"
|
||||
log.parent.mkdir(parents=True, exist_ok=True)
|
||||
env = dict(os.environ, LME_WORKSPACE_DIR=f"datasets/longmemeval/{idx}")
|
||||
started = time.strftime("%H:%M:%S")
|
||||
print(f"[start {started}] {idx}/{job}", flush=True)
|
||||
with log.open("w", encoding="utf-8") as f:
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
"reme",
|
||||
"start",
|
||||
"config=jinli_lme",
|
||||
f"job={job}",
|
||||
cwd=str(REPO),
|
||||
env=env,
|
||||
stdout=f,
|
||||
stderr=asyncio.subprocess.STDOUT,
|
||||
)
|
||||
rc = await proc.wait()
|
||||
ok = rc == 0 and job_done(idx, job)
|
||||
counters["done" if ok else "fail"] += 1
|
||||
tag = "done" if ok else "fail"
|
||||
print(f"[{tag}] {idx}/{job} rc={rc} ({counters['done']} done / {counters['fail']} fail)", flush=True)
|
||||
return ok
|
||||
|
||||
|
||||
async def run_one(idx: str, jobs: list[str], sem: asyncio.Semaphore, resume: bool, counters: dict) -> None:
|
||||
"""Run the selected jobs for one sample, serially.
|
||||
|
||||
By default every selected job is rerun (the job's own clear step wipes stale
|
||||
output first). With ``resume`` a job is skipped when its output already
|
||||
exists, so an interrupted batch can continue without redoing finished work.
|
||||
"""
|
||||
async with sem:
|
||||
for job in jobs:
|
||||
if resume and job_done(idx, job):
|
||||
counters["skip"] += 1
|
||||
print(f"[skip] {idx}/{job} (output exists)", flush=True)
|
||||
continue
|
||||
ok = await run_job(idx, job, counters)
|
||||
if not ok:
|
||||
# Later jobs depend on earlier ones; don't waste a run on a broken workspace.
|
||||
print(f"[abort] {idx}: {job} failed, skipping remaining jobs", flush=True)
|
||||
break
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Aggregation of agentic_answer results into one big JSON.
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
# Match ``session_id=abc123`` headers and ``"...session_id": "abc123"`` fields in
|
||||
# tool-result text, so we can list which sessions each search actually surfaced.
|
||||
_SID_RE = re.compile(r'session_id["\s:=]+"?([A-Za-z0-9_\-]+)')
|
||||
|
||||
|
||||
def _load_json(path: Path) -> dict:
|
||||
"""Load a JSON object, returning {} on any error."""
|
||||
try:
|
||||
with path.open(encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
return data if isinstance(data, dict) else {}
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return {}
|
||||
|
||||
|
||||
def parse_tool_calls(idx: str, session_id: str) -> list[dict]:
|
||||
"""Best-effort: parse the agent trajectory into an ordered tool-call summary.
|
||||
|
||||
Reads ``mem_session/agentscope/<session_id>.jsonl`` — the trajectory the
|
||||
agentic_answer run dumped — and pairs every ``tool_call`` (name + parsed
|
||||
args) with the ``session_id`` hits found in its ``tool_result``. Returns an
|
||||
empty list if the file is missing or unreadable (never raises).
|
||||
"""
|
||||
if not session_id:
|
||||
return []
|
||||
path = DATA / idx / "mem_session" / "agentscope" / f"{session_id}.jsonl"
|
||||
if not path.exists():
|
||||
return []
|
||||
|
||||
calls: dict[str, dict] = {}
|
||||
order: list[str] = []
|
||||
try:
|
||||
for line in path.read_text(encoding="utf-8").splitlines():
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
msg = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
for c in msg.get("content") or []:
|
||||
if not isinstance(c, dict):
|
||||
continue
|
||||
cid = c.get("id")
|
||||
if c.get("type") == "tool_call" and cid:
|
||||
try:
|
||||
args = json.loads(c.get("input") or "{}")
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
args = c.get("input")
|
||||
calls[cid] = {"name": c.get("name"), "args": args, "hit_session_ids": []}
|
||||
order.append(cid)
|
||||
elif c.get("type") == "tool_result" and cid in calls:
|
||||
text = ""
|
||||
for o in c.get("output") or []:
|
||||
if isinstance(o, dict) and isinstance(o.get("text"), str):
|
||||
text += o["text"]
|
||||
hits = list(dict.fromkeys(_SID_RE.findall(text)))
|
||||
calls[cid]["hit_session_ids"] = hits
|
||||
except OSError:
|
||||
return []
|
||||
|
||||
return [{"iter": i + 1, **calls[cid]} for i, cid in enumerate(order)]
|
||||
|
||||
|
||||
def build_record(idx: str) -> dict:
|
||||
"""Assemble one sample's aggregate record from its on-disk artifacts."""
|
||||
ws = DATA / idx
|
||||
query = _load_json(ws / "query.json")
|
||||
golden = _load_json(ws / "answer.json")
|
||||
mem = _load_json(ws / "mem_answer.json")
|
||||
|
||||
pred = str(mem.get("answer") or "").strip()
|
||||
session_id = str(mem.get("session_id") or "")
|
||||
llm_judge = mem.get("llm_judge") if isinstance(mem.get("llm_judge"), dict) else {}
|
||||
tool_calls = parse_tool_calls(idx, session_id) if mem else []
|
||||
|
||||
if not mem:
|
||||
status = "missing"
|
||||
elif not pred:
|
||||
status = "empty"
|
||||
elif "not provided" in pred.lower():
|
||||
status = "not_provided"
|
||||
else:
|
||||
status = "answered"
|
||||
|
||||
return {
|
||||
"idx": idx,
|
||||
"question_id": query.get("question_id"),
|
||||
"question_type": query.get("question_type"),
|
||||
"question": query.get("question"),
|
||||
"question_date": query.get("question_date"),
|
||||
"golden_answer": golden.get("answer"),
|
||||
"golden_answer_session_ids": golden.get("answer_session_ids"),
|
||||
"pred_answer": pred,
|
||||
"session_id": session_id,
|
||||
"status": status,
|
||||
"llm_judge": llm_judge.get("judgement"),
|
||||
"llm_judge_raw": llm_judge.get("raw_judgement"),
|
||||
"num_tool_calls": len(tool_calls),
|
||||
"tool_calls": tool_calls,
|
||||
}
|
||||
|
||||
|
||||
def write_aggregate(ids: list[str]) -> None:
|
||||
"""Aggregate every sample's agentic_answer artifacts into one big JSON."""
|
||||
records = [build_record(idx) for idx in ids]
|
||||
finished = [r for r in records if r["status"] != "missing"]
|
||||
by_status: dict[str, int] = {}
|
||||
by_llm_judge: dict[str, int] = {}
|
||||
for r in records:
|
||||
by_status[r["status"]] = by_status.get(r["status"], 0) + 1
|
||||
judgement = r.get("llm_judge") or "missing"
|
||||
by_llm_judge[judgement] = by_llm_judge.get(judgement, 0) + 1
|
||||
|
||||
payload = {
|
||||
"generated_at": time.strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"total": len(records),
|
||||
"finished": len(finished),
|
||||
"by_status": by_status,
|
||||
"by_llm_judge": by_llm_judge,
|
||||
"samples": records,
|
||||
}
|
||||
AGGREGATE.parent.mkdir(parents=True, exist_ok=True)
|
||||
AGGREGATE.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(f"[aggregate] wrote {len(records)} samples ({len(finished)} finished) -> {AGGREGATE}", flush=True)
|
||||
|
||||
|
||||
async def main() -> int:
|
||||
"""Run the driver."""
|
||||
args = parse_args()
|
||||
LOGDIR.mkdir(parents=True, exist_ok=True)
|
||||
jobs = selected_jobs(args.job)
|
||||
|
||||
ids = sample_ids()
|
||||
if args.end and args.end < args.start:
|
||||
raise ValueError(f"--end ({args.end}) must be >= --start ({args.start})")
|
||||
ids = [i for i in ids if int(i) >= args.start and (not args.end or int(i) <= args.end)]
|
||||
if args.limit:
|
||||
ids = ids[: args.limit]
|
||||
|
||||
# Without --resume every job reruns; with --resume, jobs whose output exists are skipped.
|
||||
def todo_jobs(i: str) -> list[str]:
|
||||
return [j for j in jobs if not (args.resume and job_done(i, j))]
|
||||
|
||||
pending = [i for i in ids if todo_jobs(i)]
|
||||
print(
|
||||
f"jobs={jobs} resume={args.resume} samples total={len(ids)} pending={len(pending)} "
|
||||
f"concurrency={args.concurrency} stagger={args.stagger}s",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
if args.dry_run:
|
||||
for i in pending:
|
||||
print(f"[would-run] {i}: {todo_jobs(i)}")
|
||||
return 0
|
||||
|
||||
sem = asyncio.Semaphore(args.concurrency)
|
||||
counters = {"done": 0, "fail": 0, "skip": 0}
|
||||
tasks: list[asyncio.Task] = []
|
||||
for n, idx in enumerate(ids):
|
||||
if n and args.stagger > 0:
|
||||
await asyncio.sleep(args.stagger) # stagger each launch relative to the previous
|
||||
tasks.append(asyncio.create_task(run_one(idx, jobs, sem, args.resume, counters)))
|
||||
|
||||
await asyncio.gather(*tasks, return_exceptions=True)
|
||||
print(
|
||||
f"ALL FINISHED done={counters['done']} fail={counters['fail']} skip={counters['skip']}",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
if any(j in jobs for j in ("agentic_answer", "llm_judge")) and not args.no_aggregate:
|
||||
write_aggregate(ids)
|
||||
|
||||
return 0 if counters["fail"] == 0 else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(asyncio.run(main()))
|
||||
|
|
@ -1,469 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Review every LongMemEval golden answer with the configured Claude Code job.
|
||||
|
||||
Every numeric ``datasets/longmemeval/<idx>`` workspace is processed sequentially.
|
||||
The reference JSONL files are merged by ``question_id`` and supplied only when
|
||||
they contain an alternative answer for that sample:
|
||||
|
||||
reme start config=jinli_lme job=final_answer_review
|
||||
|
||||
The job returns a plain four-field JSON object with ``reason``,
|
||||
``golden_answer_correct``, ``answer``, and ``is_session_time_wrong``. After
|
||||
every new success, this driver atomically rewrites the complete accumulated
|
||||
output JSONL so an interrupted run can safely resume.
|
||||
|
||||
Examples:
|
||||
python benchmark/longmemeval/run_final_answer_review.py
|
||||
python benchmark/longmemeval/run_final_answer_review.py --exclude-reference-question-ids
|
||||
python benchmark/longmemeval/run_final_answer_review.py --only-reference-question-ids --rerun-selected
|
||||
python benchmark/longmemeval/run_final_answer_review.py --concurrency 2 --submit-interval-seconds 6
|
||||
python benchmark/longmemeval/run_final_answer_review.py --question-id e47becba
|
||||
python benchmark/longmemeval/run_final_answer_review.py --reference path/to/results.jsonl
|
||||
python benchmark/longmemeval/run_final_answer_review.py --limit 3
|
||||
python benchmark/longmemeval/run_final_answer_review.py --no-resume
|
||||
python benchmark/longmemeval/run_final_answer_review.py --dry-run
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import concurrent.futures
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
DATA = REPO / "datasets" / "longmemeval"
|
||||
DEFAULT_REFERENCES = (
|
||||
REPO / "benchmark" / "longmemeval" / "golden_check_list_false.jsonl",
|
||||
REPO / "benchmark" / "longmemeval" / "merge_confirm_jinli_false.jsonl",
|
||||
)
|
||||
DEFAULT_OUTPUT = REPO / "benchmark" / "longmemeval" / "final_answer_review.jsonl"
|
||||
DEFAULT_LOG_DIR = REPO / "logs" / "final_answer_review"
|
||||
REFERENCE_PATHS_ENV = "LME_FINAL_ANSWER_REFERENCE_PATHS"
|
||||
MAX_CONCURRENCY = 3
|
||||
MIN_SUBMIT_INTERVAL_SECONDS = 5.0
|
||||
DEFAULT_SUBMIT_INTERVAL_SECONDS = 5.1
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
"""Parse command-line arguments."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--question-id",
|
||||
dest="question_ids",
|
||||
action="append",
|
||||
help="process only this dataset question ID; repeat for multiple IDs (default: all)",
|
||||
)
|
||||
reference_selection = parser.add_mutually_exclusive_group()
|
||||
reference_selection.add_argument(
|
||||
"--exclude-reference-question-ids",
|
||||
action="store_true",
|
||||
help="skip question IDs found in the selected reference-answer JSONL files",
|
||||
)
|
||||
reference_selection.add_argument(
|
||||
"--only-reference-question-ids",
|
||||
action="store_true",
|
||||
help="process only question IDs found in the selected reference-answer JSONL files",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--reference",
|
||||
dest="references",
|
||||
action="append",
|
||||
type=Path,
|
||||
help="reference-answer JSONL; repeat for multiple files (default: built-in disputed results)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
type=Path,
|
||||
default=DEFAULT_OUTPUT,
|
||||
help=f"output JSONL (default: {DEFAULT_OUTPUT})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--log-dir",
|
||||
type=Path,
|
||||
default=DEFAULT_LOG_DIR,
|
||||
help="directory for per-question logs",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--concurrency",
|
||||
type=int,
|
||||
default=MAX_CONCURRENCY,
|
||||
help=f"maximum concurrent jobs, from 1 to {MAX_CONCURRENCY} (default: {MAX_CONCURRENCY})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--submit-interval-seconds",
|
||||
type=float,
|
||||
default=DEFAULT_SUBMIT_INTERVAL_SECONDS,
|
||||
help=f"minimum time between job submissions; must be > {MIN_SUBMIT_INTERVAL_SECONDS:g} "
|
||||
f"(default: {DEFAULT_SUBMIT_INTERVAL_SECONDS:g})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--limit",
|
||||
type=int,
|
||||
default=0,
|
||||
help="process only the first N pending questions (0 = all)",
|
||||
)
|
||||
resume_mode = parser.add_mutually_exclusive_group()
|
||||
resume_mode.add_argument(
|
||||
"--no-resume",
|
||||
action="store_true",
|
||||
help="ignore existing output and rerun every selected question",
|
||||
)
|
||||
resume_mode.add_argument(
|
||||
"--rerun-selected",
|
||||
action="store_true",
|
||||
help="rerun every selected question while preserving existing results until replacements finish",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dry-run",
|
||||
action="store_true",
|
||||
help="show the selected cases without invoking ReMe",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def _read_jsonl(path: Path) -> list[dict[str, Any]]:
|
||||
"""Read a JSONL file and reject malformed or non-object rows."""
|
||||
rows: list[dict[str, Any]] = []
|
||||
try:
|
||||
with path.open(encoding="utf-8") as file:
|
||||
for line_number, line in enumerate(file, start=1):
|
||||
if not line.strip():
|
||||
continue
|
||||
try:
|
||||
row = json.loads(line)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError(f"Invalid JSON at {path}:{line_number}") from exc
|
||||
if not isinstance(row, dict):
|
||||
raise ValueError(f"Expected a JSON object at {path}:{line_number}")
|
||||
rows.append(row)
|
||||
except OSError as exc:
|
||||
raise FileNotFoundError(f"Cannot read JSONL file: {path}") from exc
|
||||
return rows
|
||||
|
||||
|
||||
def merge_references(paths: list[Path]) -> dict[str, list[dict[str, Any]]]:
|
||||
"""Merge reference rows by question ID, preserving file and row order."""
|
||||
merged: dict[str, list[dict[str, Any]]] = {}
|
||||
seen_sources: set[tuple[str, str]] = set()
|
||||
for path in paths:
|
||||
for row in _read_jsonl(path):
|
||||
question_id = str(row.get("question_id") or "").strip()
|
||||
if not question_id:
|
||||
raise ValueError(f"Reference row in {path} has no question_id")
|
||||
source_key = (question_id, str(path.resolve()))
|
||||
if source_key in seen_sources:
|
||||
raise ValueError(f"Duplicate question_id={question_id!r} within {path}")
|
||||
seen_sources.add(source_key)
|
||||
merged.setdefault(question_id, []).append({"source": path.name, **row})
|
||||
if not merged:
|
||||
raise ValueError("No reference answers found")
|
||||
return merged
|
||||
|
||||
|
||||
def workspace_map() -> dict[str, Path]:
|
||||
"""Map every dataset question ID to its numeric sample workspace."""
|
||||
mapping: dict[str, Path] = {}
|
||||
for workspace in sorted(
|
||||
(path for path in DATA.iterdir() if path.is_dir() and path.name.isdigit()),
|
||||
key=lambda p: int(p.name),
|
||||
):
|
||||
query_path = workspace / "query.json"
|
||||
if not query_path.is_file():
|
||||
continue
|
||||
try:
|
||||
with query_path.open(encoding="utf-8") as file:
|
||||
query = json.load(file)
|
||||
except (OSError, json.JSONDecodeError) as exc:
|
||||
raise ValueError(f"Cannot parse {query_path}") from exc
|
||||
if not isinstance(query, dict):
|
||||
raise ValueError(f"Expected a JSON object in {query_path}")
|
||||
question_id = str(query.get("question_id") or "").strip()
|
||||
if not question_id:
|
||||
raise ValueError(f"Missing question_id in {query_path}")
|
||||
if question_id in mapping:
|
||||
raise ValueError(
|
||||
f"Duplicate dataset question_id={question_id!r}: {mapping[question_id]} and {workspace}",
|
||||
)
|
||||
mapping[question_id] = workspace
|
||||
return mapping
|
||||
|
||||
|
||||
def select_question_ids(
|
||||
mapping: dict[str, Path],
|
||||
requested: list[str] | None,
|
||||
excluded: set[str] | None = None,
|
||||
) -> list[str]:
|
||||
"""Return all dataset IDs or validate an explicitly requested subset."""
|
||||
excluded = excluded or set()
|
||||
if not requested:
|
||||
return [question_id for question_id in mapping if question_id not in excluded]
|
||||
selected: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for raw_question_id in requested:
|
||||
question_id = raw_question_id.strip()
|
||||
if not question_id:
|
||||
raise ValueError("--question-id must not be empty")
|
||||
if question_id in seen:
|
||||
raise ValueError(f"Duplicate --question-id: {question_id}")
|
||||
if question_id not in mapping:
|
||||
raise ValueError(f"No dataset workspace for question ID: {question_id}")
|
||||
if question_id not in excluded:
|
||||
selected.append(question_id)
|
||||
seen.add(question_id)
|
||||
return selected
|
||||
|
||||
|
||||
def _validate_result(value: Any, *, source: str) -> dict[str, Any]:
|
||||
"""Validate the final four-field answer contract."""
|
||||
expected_keys = {"reason", "golden_answer_correct", "answer", "is_session_time_wrong"}
|
||||
if not isinstance(value, dict) or set(value) != expected_keys:
|
||||
raise ValueError(
|
||||
f"{source} must contain exactly 'reason', 'golden_answer_correct', 'answer', "
|
||||
"and 'is_session_time_wrong'",
|
||||
)
|
||||
if not isinstance(value["reason"], str) or not value["reason"].strip():
|
||||
raise ValueError(f"{source} has an invalid reason")
|
||||
if not isinstance(value["golden_answer_correct"], bool):
|
||||
raise ValueError(f"{source} has an invalid golden_answer_correct")
|
||||
if not isinstance(value["answer"], str):
|
||||
raise ValueError(f"{source} has an invalid answer")
|
||||
answer = value["answer"].strip()
|
||||
if value["golden_answer_correct"] and answer:
|
||||
raise ValueError(f"{source} answer must be empty when golden_answer_correct is true")
|
||||
if not value["golden_answer_correct"] and not answer:
|
||||
raise ValueError(f"{source} answer must be non-empty when golden_answer_correct is false")
|
||||
if not isinstance(value["is_session_time_wrong"], bool):
|
||||
raise ValueError(f"{source} has an invalid is_session_time_wrong")
|
||||
return {
|
||||
"reason": value["reason"].strip(),
|
||||
"golden_answer_correct": value["golden_answer_correct"],
|
||||
"answer": answer,
|
||||
"is_session_time_wrong": False,
|
||||
}
|
||||
|
||||
|
||||
def load_existing(path: Path) -> dict[str, dict[str, Any]]:
|
||||
"""Load resumable output, rejecting duplicate or malformed rows."""
|
||||
if not path.exists():
|
||||
return {}
|
||||
results: dict[str, dict[str, Any]] = {}
|
||||
for row in _read_jsonl(path):
|
||||
question_id = str(row.get("question_id") or "").strip()
|
||||
if not question_id:
|
||||
raise ValueError(f"Existing output row in {path} has no question_id")
|
||||
if question_id in results:
|
||||
raise ValueError(
|
||||
f"Duplicate question_id={question_id!r} in existing output {path}",
|
||||
)
|
||||
results[question_id] = _validate_result(
|
||||
{key: value for key, value in row.items() if key != "question_id"},
|
||||
source=f"existing result for {question_id}",
|
||||
)
|
||||
return results
|
||||
|
||||
|
||||
def atomic_write_results(
|
||||
path: Path,
|
||||
order: list[str],
|
||||
results: dict[str, dict[str, Any]],
|
||||
) -> None:
|
||||
"""Atomically rewrite all accumulated rows in stable merged-input order."""
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temp_path: Path | None = None
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(
|
||||
"w",
|
||||
encoding="utf-8",
|
||||
dir=path.parent,
|
||||
prefix=f".{path.name}.",
|
||||
delete=False,
|
||||
) as file:
|
||||
temp_path = Path(file.name)
|
||||
for question_id in order:
|
||||
if question_id not in results:
|
||||
continue
|
||||
row = {"question_id": question_id, **results[question_id]}
|
||||
file.write(
|
||||
json.dumps(row, ensure_ascii=False, separators=(",", ":")) + "\n",
|
||||
)
|
||||
file.flush()
|
||||
os.fsync(file.fileno())
|
||||
os.replace(temp_path, path)
|
||||
finally:
|
||||
if temp_path is not None and temp_path.exists():
|
||||
temp_path.unlink()
|
||||
|
||||
|
||||
def run_one(
|
||||
question_id: str,
|
||||
workspace: Path,
|
||||
log_dir: Path,
|
||||
reference_paths: list[Path],
|
||||
) -> dict[str, Any]:
|
||||
"""Run the configured one-shot job and validate its stdout JSON."""
|
||||
env = dict(os.environ, LME_WORKSPACE_DIR=str(workspace.relative_to(REPO)))
|
||||
env[REFERENCE_PATHS_ENV] = json.dumps(
|
||||
[str(path.resolve()) for path in reference_paths],
|
||||
ensure_ascii=False,
|
||||
)
|
||||
completed = subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
"-c",
|
||||
"from reme.reme import main; main()",
|
||||
"start",
|
||||
"config=jinli_lme",
|
||||
"job=final_answer_review",
|
||||
],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
text=True,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
check=False,
|
||||
)
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
log_path = log_dir / f"{question_id}.log"
|
||||
log_text = (
|
||||
f"workspace={workspace}\nreturncode={completed.returncode}\n\n"
|
||||
f"[stdout]\n{completed.stdout}\n[stderr]\n{completed.stderr}"
|
||||
)
|
||||
log_path.write_text(
|
||||
log_text,
|
||||
encoding="utf-8",
|
||||
)
|
||||
if completed.returncode != 0:
|
||||
raise RuntimeError(
|
||||
f"Job failed for {question_id} with rc={completed.returncode}; see {log_path}",
|
||||
)
|
||||
try:
|
||||
value = json.loads(completed.stdout.strip())
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError(
|
||||
f"Job stdout is not JSON for {question_id}; see {log_path}",
|
||||
) from exc
|
||||
return _validate_result(value, source=f"job result for {question_id}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Review and checkpoint the selected dataset cases sequentially."""
|
||||
args = parse_args()
|
||||
if args.limit < 0:
|
||||
raise ValueError("--limit must be >= 0")
|
||||
if not 1 <= args.concurrency <= MAX_CONCURRENCY:
|
||||
raise ValueError(f"--concurrency must be between 1 and {MAX_CONCURRENCY}")
|
||||
if args.submit_interval_seconds <= MIN_SUBMIT_INTERVAL_SECONDS:
|
||||
raise ValueError(
|
||||
f"--submit-interval-seconds must be > {MIN_SUBMIT_INTERVAL_SECONDS:g}",
|
||||
)
|
||||
|
||||
reference_paths = [path.resolve() for path in (args.references or DEFAULT_REFERENCES)]
|
||||
mapping = workspace_map()
|
||||
references = merge_references(reference_paths)
|
||||
missing = [question_id for question_id in references if question_id not in mapping]
|
||||
if missing:
|
||||
raise ValueError(f"No dataset workspace for question IDs: {', '.join(missing)}")
|
||||
|
||||
full_order = list(mapping)
|
||||
excluded = set(references) if args.exclude_reference_question_ids else set()
|
||||
order = select_question_ids(mapping, args.question_ids, excluded)
|
||||
if args.only_reference_question_ids:
|
||||
order = [question_id for question_id in order if question_id in references]
|
||||
results = {} if args.no_resume else load_existing(args.output.resolve())
|
||||
pending = (
|
||||
list(order) if args.rerun_selected else [question_id for question_id in order if question_id not in results]
|
||||
)
|
||||
if args.limit:
|
||||
pending = pending[: args.limit]
|
||||
|
||||
no_reference = sum(question_id not in references for question_id in order)
|
||||
one_reference = sum(len(references.get(question_id, [])) == 1 for question_id in order)
|
||||
multiple_references = sum(len(references.get(question_id, [])) > 1 for question_id in order)
|
||||
print(
|
||||
f"total={len(order)} no_reference={no_reference} one_reference={one_reference} "
|
||||
f"multiple_references={multiple_references} "
|
||||
f"excluded={len(excluded)} "
|
||||
f"only_reference_questions={args.only_reference_question_ids} "
|
||||
f"concurrency={args.concurrency} submit_interval={args.submit_interval_seconds:g}s "
|
||||
f"existing={len(results)} pending={len(pending)} output={args.output.resolve()}",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
if args.dry_run:
|
||||
for question_id in pending:
|
||||
print(
|
||||
f"[would-run] question_id={question_id} workspace={mapping[question_id].name} "
|
||||
f"references={len(references.get(question_id, []))}",
|
||||
)
|
||||
return 0
|
||||
|
||||
executor = concurrent.futures.ThreadPoolExecutor(max_workers=args.concurrency)
|
||||
active: dict[concurrent.futures.Future[dict[str, Any]], tuple[int, str]] = {}
|
||||
next_position = 0
|
||||
saved_count = 0
|
||||
next_submit_at = 0.0
|
||||
try:
|
||||
while next_position < len(pending) or active:
|
||||
can_submit = next_position < len(pending) and len(active) < args.concurrency
|
||||
if can_submit and time.monotonic() >= next_submit_at:
|
||||
question_id = pending[next_position]
|
||||
position = next_position + 1
|
||||
workspace = mapping[question_id]
|
||||
print(
|
||||
f"[submit {position}/{len(pending)}] question_id={question_id} "
|
||||
f"workspace={workspace.name} references={len(references.get(question_id, []))}",
|
||||
flush=True,
|
||||
)
|
||||
future = executor.submit(
|
||||
run_one,
|
||||
question_id,
|
||||
workspace,
|
||||
args.log_dir.resolve(),
|
||||
reference_paths,
|
||||
)
|
||||
active[future] = (position, question_id)
|
||||
next_position += 1
|
||||
next_submit_at = time.monotonic() + args.submit_interval_seconds
|
||||
continue
|
||||
|
||||
if not active:
|
||||
time.sleep(max(0.0, next_submit_at - time.monotonic()))
|
||||
continue
|
||||
|
||||
timeout = None
|
||||
if can_submit:
|
||||
timeout = max(0.0, next_submit_at - time.monotonic())
|
||||
done, _ = concurrent.futures.wait(
|
||||
active,
|
||||
timeout=timeout,
|
||||
return_when=concurrent.futures.FIRST_COMPLETED,
|
||||
)
|
||||
for future in done:
|
||||
position, question_id = active.pop(future)
|
||||
results[question_id] = future.result()
|
||||
atomic_write_results(args.output.resolve(), full_order, results)
|
||||
saved_count += 1
|
||||
print(
|
||||
f"[saved {saved_count}/{len(pending)}] submitted_position={position} " f"question_id={question_id}",
|
||||
flush=True,
|
||||
)
|
||||
finally:
|
||||
executor.shutdown(wait=True, cancel_futures=True)
|
||||
|
||||
print(
|
||||
f"ALL FINISHED total_saved={sum(question_id in results for question_id in order)}",
|
||||
flush=True,
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
|
@ -1,216 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Run LongMemEval ``golden_check`` concurrently across samples.
|
||||
|
||||
For every workspace under ``datasets/longmemeval/<idx>`` in the selected numeric
|
||||
range, this launches:
|
||||
|
||||
reme start config=jinli_lme job=golden_check
|
||||
|
||||
with ``LME_WORKSPACE_DIR`` pointed at that sample. Multiple samples can run at
|
||||
once, capped by ``--concurrency``. The ``golden_check`` job itself waits for
|
||||
``session_review.json`` when configured with ``wait_for_paths_step`` in
|
||||
``jinli_lme.yaml``. Each sample's stdout/stderr goes to
|
||||
``logs/golden_check/<idx>.log``.
|
||||
|
||||
By default the script processes samples 0..499 inclusive and reruns every sample
|
||||
in that range. Pass ``--resume`` to skip samples whose ``check_golden.json``
|
||||
already exists.
|
||||
|
||||
Examples:
|
||||
python benchmark/longmemeval/run_golden_check.py
|
||||
python benchmark/longmemeval/run_golden_check.py --start 187 --end 499
|
||||
python benchmark/longmemeval/run_golden_check.py --concurrency 8 --stagger 1
|
||||
python benchmark/longmemeval/run_golden_check.py --progress-interval 10
|
||||
python benchmark/longmemeval/run_golden_check.py --resume
|
||||
python benchmark/longmemeval/run_golden_check.py --limit 5 --dry-run
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
DATA = REPO / "datasets" / "longmemeval"
|
||||
LOGDIR = REPO / "logs" / "golden_check"
|
||||
OUTPUT_FILENAME = "check_golden.json"
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
"""Parse command-line arguments."""
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--start", type=int, default=0, help="first numeric sample id to process, inclusive (default 0)")
|
||||
p.add_argument("--end", type=int, default=499, help="last numeric sample id to process, inclusive (default 499)")
|
||||
p.add_argument("--limit", type=int, default=0, help="only process the first N selected samples (0 = all)")
|
||||
p.add_argument("--concurrency", type=int, default=3, help="max samples running at once (default 3)")
|
||||
p.add_argument("--stagger", type=float, default=1.0, help="seconds between consecutive launches (default 1)")
|
||||
p.add_argument(
|
||||
"--progress-interval",
|
||||
type=float,
|
||||
default=30.0,
|
||||
help="seconds between progress reports while running (0 = disabled, default 30)",
|
||||
)
|
||||
p.add_argument(
|
||||
"--resume",
|
||||
action="store_true",
|
||||
help=f"skip samples whose {OUTPUT_FILENAME} already exists",
|
||||
)
|
||||
p.add_argument("--dry-run", action="store_true", help="list what would run, launch nothing")
|
||||
return p.parse_args()
|
||||
|
||||
|
||||
def sample_ids() -> list[str]:
|
||||
"""List all sample IDs (numeric workspace dirs), numerically sorted."""
|
||||
ids = [p.name for p in DATA.iterdir() if p.is_dir() and p.name.isdigit()]
|
||||
return sorted(ids, key=int)
|
||||
|
||||
|
||||
def output_is_current(idx: str) -> bool:
|
||||
"""Return True when the sample already has a current-schema golden-check artifact."""
|
||||
path = DATA / idx / OUTPUT_FILENAME
|
||||
try:
|
||||
with path.open(encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return False
|
||||
verdict = data.get("verdict") if isinstance(data, dict) else None
|
||||
if not isinstance(verdict, dict):
|
||||
return False
|
||||
return isinstance(verdict.get("golden_answer_correct"), bool) and isinstance(
|
||||
verdict.get("answer_session_ids_correct"),
|
||||
bool,
|
||||
)
|
||||
|
||||
|
||||
def print_progress(counters: dict, active: set[str], selected_total: int, started_at: float) -> None:
|
||||
"""Print a one-line progress snapshot."""
|
||||
finished = counters["done"] + counters["fail"] + counters["skip"]
|
||||
running = len(active)
|
||||
outstanding = max(selected_total - finished - running, 0)
|
||||
elapsed = time.monotonic() - started_at
|
||||
print(
|
||||
f"[progress] selected={selected_total} done={counters['done']} fail={counters['fail']} "
|
||||
f"skip={counters['skip']} running={running} outstanding={outstanding} "
|
||||
f"elapsed={elapsed:.0f}s",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
|
||||
async def progress_reporter(
|
||||
counters: dict,
|
||||
active: set[str],
|
||||
selected_total: int,
|
||||
started_at: float,
|
||||
interval: float,
|
||||
stop: asyncio.Event,
|
||||
) -> None:
|
||||
"""Periodically report progress until ``stop`` is set."""
|
||||
if interval <= 0:
|
||||
return
|
||||
while not stop.is_set():
|
||||
try:
|
||||
await asyncio.wait_for(stop.wait(), timeout=interval)
|
||||
except asyncio.TimeoutError:
|
||||
print_progress(counters, active, selected_total, started_at)
|
||||
|
||||
|
||||
async def run_one(idx: str, sem: asyncio.Semaphore, resume: bool, counters: dict, active: set[str]) -> None:
|
||||
"""Run ``golden_check`` for one sample."""
|
||||
if resume and output_is_current(idx):
|
||||
counters["skip"] += 1
|
||||
print(f"[skip] {idx} ({OUTPUT_FILENAME} exists)", flush=True)
|
||||
return
|
||||
|
||||
async with sem:
|
||||
active.add(idx)
|
||||
log = LOGDIR / f"{idx}.log"
|
||||
log.parent.mkdir(parents=True, exist_ok=True)
|
||||
env = dict(os.environ, LME_WORKSPACE_DIR=f"datasets/longmemeval/{idx}")
|
||||
|
||||
started = time.strftime("%H:%M:%S")
|
||||
print(f"[start {started}] {idx}", flush=True)
|
||||
try:
|
||||
with log.open("w", encoding="utf-8") as f:
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
"reme",
|
||||
"start",
|
||||
"config=jinli_lme",
|
||||
"job=golden_check",
|
||||
cwd=str(REPO),
|
||||
env=env,
|
||||
stdout=f,
|
||||
stderr=asyncio.subprocess.STDOUT,
|
||||
)
|
||||
rc = await proc.wait()
|
||||
|
||||
ok = rc == 0 and output_is_current(idx)
|
||||
counters["done" if ok else "fail"] += 1
|
||||
tag = "done" if ok else "fail"
|
||||
print(
|
||||
f"[{tag}] {idx} rc={rc} log={log} ({counters['done']} done / {counters['fail']} fail)",
|
||||
flush=True,
|
||||
)
|
||||
finally:
|
||||
active.discard(idx)
|
||||
|
||||
|
||||
async def main() -> int:
|
||||
"""Run the concurrent driver."""
|
||||
args = parse_args()
|
||||
if args.end < args.start:
|
||||
raise ValueError(f"--end ({args.end}) must be >= --start ({args.start})")
|
||||
if args.concurrency < 1:
|
||||
raise ValueError("--concurrency must be >= 1")
|
||||
if args.progress_interval < 0:
|
||||
raise ValueError("--progress-interval must be >= 0")
|
||||
|
||||
LOGDIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
ids = [i for i in sample_ids() if args.start <= int(i) <= args.end]
|
||||
if args.limit:
|
||||
ids = ids[: args.limit]
|
||||
|
||||
pending = [i for i in ids if not (args.resume and output_is_current(i))]
|
||||
print(
|
||||
f"job=golden_check samples total={len(ids)} pending={len(pending)} "
|
||||
f"range={args.start}..{args.end} resume={args.resume} "
|
||||
f"concurrency={args.concurrency} stagger={args.stagger}s",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
if args.dry_run:
|
||||
for idx in pending:
|
||||
print(f"[would-run] {idx}")
|
||||
return 0
|
||||
|
||||
sem = asyncio.Semaphore(args.concurrency)
|
||||
counters = {"done": 0, "fail": 0, "skip": 0}
|
||||
active: set[str] = set()
|
||||
started_at = time.monotonic()
|
||||
stop_progress = asyncio.Event()
|
||||
progress_task = asyncio.create_task(
|
||||
progress_reporter(counters, active, len(ids), started_at, args.progress_interval, stop_progress),
|
||||
)
|
||||
tasks: list[asyncio.Task] = []
|
||||
try:
|
||||
for n, idx in enumerate(ids):
|
||||
if n and args.stagger > 0:
|
||||
await asyncio.sleep(args.stagger)
|
||||
tasks.append(asyncio.create_task(run_one(idx, sem, args.resume, counters, active)))
|
||||
|
||||
await asyncio.gather(*tasks)
|
||||
finally:
|
||||
stop_progress.set()
|
||||
await progress_task
|
||||
print_progress(counters, active, len(ids), started_at)
|
||||
print(
|
||||
f"ALL FINISHED done={counters['done']} fail={counters['fail']} skip={counters['skip']}",
|
||||
flush=True,
|
||||
)
|
||||
return 0 if counters["fail"] == 0 else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(asyncio.run(main()))
|
||||
|
|
@ -1,203 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Run LongMemEval ``session_review`` concurrently across samples.
|
||||
|
||||
For every workspace under ``datasets/longmemeval/<idx>`` in the selected numeric
|
||||
range, this launches:
|
||||
|
||||
reme start config=jinli_lme job=session_review
|
||||
|
||||
with ``LME_WORKSPACE_DIR`` pointed at that sample. Multiple samples can run at
|
||||
once, capped by ``--concurrency``. By default this runner launches one sample at
|
||||
a time; request submission is throttled inside each ``session_review`` process.
|
||||
Each sample's stdout/stderr goes to ``logs/session_review/<idx>.log``.
|
||||
|
||||
By default the script processes samples 0..499 inclusive and reruns every sample
|
||||
in that range. Pass ``--resume`` to skip samples whose ``session_review.json``
|
||||
already exists.
|
||||
|
||||
Examples:
|
||||
python benchmark/longmemeval/run_session_review.py
|
||||
python benchmark/longmemeval/run_session_review.py --start 187 --end 499
|
||||
python benchmark/longmemeval/run_session_review.py --concurrency 2
|
||||
python benchmark/longmemeval/run_session_review.py --resume
|
||||
python benchmark/longmemeval/run_session_review.py --limit 5 --dry-run
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
DATA = REPO / "datasets" / "longmemeval"
|
||||
LOGDIR = REPO / "logs" / "session_review"
|
||||
OUTPUT_FILENAME = "session_review.json"
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
"""Parse command-line arguments."""
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--start", type=int, default=0, help="first numeric sample id to process, inclusive (default 0)")
|
||||
p.add_argument("--end", type=int, default=499, help="last numeric sample id to process, inclusive (default 499)")
|
||||
p.add_argument("--limit", type=int, default=0, help="only process the first N selected samples (0 = all)")
|
||||
p.add_argument("--concurrency", type=int, default=1, help="max samples running at once (default 1)")
|
||||
p.add_argument("--stagger", type=float, default=1.0, help="seconds between worker launches (default 1)")
|
||||
p.add_argument(
|
||||
"--resume",
|
||||
action="store_true",
|
||||
help=f"skip samples whose {OUTPUT_FILENAME} already exists",
|
||||
)
|
||||
p.add_argument("--dry-run", action="store_true", help="list what would run, launch nothing")
|
||||
p.add_argument("--stop-on-fail", action="store_true", help="stop immediately after the first failed sample")
|
||||
return p.parse_args()
|
||||
|
||||
|
||||
def sample_ids() -> list[str]:
|
||||
"""List all sample IDs (numeric workspace dirs), numerically sorted."""
|
||||
ids = [p.name for p in DATA.iterdir() if p.is_dir() and p.name.isdigit()]
|
||||
return sorted(ids, key=int)
|
||||
|
||||
|
||||
def output_exists(idx: str) -> bool:
|
||||
"""Return True when the sample already has a session review artifact."""
|
||||
return (DATA / idx / OUTPUT_FILENAME).exists()
|
||||
|
||||
|
||||
def output_is_healthy(idx: str) -> bool:
|
||||
"""Return True when ``session_review.json`` exists and has no failed reviews."""
|
||||
path = DATA / idx / OUTPUT_FILENAME
|
||||
if not path.exists():
|
||||
return False
|
||||
try:
|
||||
with path.open(encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return False
|
||||
review = data.get("review") if isinstance(data, dict) else None
|
||||
if not isinstance(review, dict):
|
||||
return False
|
||||
raw = review.get("num_failed_reviews")
|
||||
if isinstance(raw, int):
|
||||
return raw == 0
|
||||
failed_reviews = review.get("failed_reviews")
|
||||
return not failed_reviews
|
||||
|
||||
|
||||
async def run_one(idx: str, active: set[str]) -> bool:
|
||||
"""Run ``session_review`` for one sample. Returns True on success."""
|
||||
log = LOGDIR / f"{idx}.log"
|
||||
log.parent.mkdir(parents=True, exist_ok=True)
|
||||
env = dict(os.environ, LME_WORKSPACE_DIR=f"datasets/longmemeval/{idx}")
|
||||
|
||||
started = time.strftime("%H:%M:%S")
|
||||
print(f"[start {started}] {idx}", flush=True)
|
||||
active.add(idx)
|
||||
try:
|
||||
with log.open("w", encoding="utf-8") as f:
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
"reme",
|
||||
"start",
|
||||
"config=jinli_lme",
|
||||
"job=session_review",
|
||||
cwd=str(REPO),
|
||||
env=env,
|
||||
stdout=f,
|
||||
stderr=asyncio.subprocess.STDOUT,
|
||||
)
|
||||
rc = await proc.wait()
|
||||
finally:
|
||||
active.discard(idx)
|
||||
|
||||
ok = rc == 0 and output_exists(idx)
|
||||
tag = "done" if ok else "fail"
|
||||
print(f"[{tag}] {idx} rc={rc} log={log}", flush=True)
|
||||
return ok
|
||||
|
||||
|
||||
async def worker(
|
||||
name: int,
|
||||
queue: asyncio.Queue[str],
|
||||
args: argparse.Namespace,
|
||||
counters: dict[str, int],
|
||||
active: set[str],
|
||||
stop: asyncio.Event,
|
||||
) -> None:
|
||||
"""Run samples from ``queue`` until exhausted or fail-fast is triggered."""
|
||||
if name and args.stagger > 0:
|
||||
await asyncio.sleep(args.stagger * name)
|
||||
|
||||
while not stop.is_set():
|
||||
try:
|
||||
idx = queue.get_nowait()
|
||||
except asyncio.QueueEmpty:
|
||||
return
|
||||
|
||||
try:
|
||||
if args.resume and output_is_healthy(idx):
|
||||
counters["skip"] += 1
|
||||
print(f"[skip] {idx} (healthy {OUTPUT_FILENAME} exists)", flush=True)
|
||||
continue
|
||||
|
||||
if await run_one(idx, active):
|
||||
counters["done"] += 1
|
||||
else:
|
||||
counters["fail"] += 1
|
||||
if args.stop_on_fail:
|
||||
stop.set()
|
||||
finally:
|
||||
queue.task_done()
|
||||
|
||||
|
||||
async def main() -> int:
|
||||
"""Run the concurrent driver."""
|
||||
args = parse_args()
|
||||
if args.end < args.start:
|
||||
raise ValueError(f"--end ({args.end}) must be >= --start ({args.start})")
|
||||
if args.concurrency < 1:
|
||||
raise ValueError("--concurrency must be >= 1")
|
||||
if args.stagger < 0:
|
||||
raise ValueError("--stagger must be >= 0")
|
||||
|
||||
LOGDIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
ids = [i for i in sample_ids() if args.start <= int(i) <= args.end]
|
||||
if args.limit:
|
||||
ids = ids[: args.limit]
|
||||
|
||||
pending = [i for i in ids if not (args.resume and output_exists(i))]
|
||||
print(
|
||||
f"job=session_review samples total={len(ids)} pending={len(pending)} "
|
||||
f"range={args.start}..{args.end} resume={args.resume} "
|
||||
f"concurrency={args.concurrency} stagger={args.stagger}s",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
if args.dry_run:
|
||||
for idx in pending:
|
||||
print(f"[would-run] {idx}")
|
||||
return 0
|
||||
|
||||
counters: dict[str, int] = {"done": 0, "fail": 0, "skip": 0}
|
||||
active: set[str] = set()
|
||||
stop = asyncio.Event()
|
||||
queue: asyncio.Queue[str] = asyncio.Queue()
|
||||
for idx in ids:
|
||||
queue.put_nowait(idx)
|
||||
|
||||
workers = [
|
||||
asyncio.create_task(worker(n, queue, args, counters, active, stop))
|
||||
for n in range(min(args.concurrency, len(ids)))
|
||||
]
|
||||
await asyncio.gather(*workers)
|
||||
|
||||
print(
|
||||
f"ALL FINISHED done={counters['done']} fail={counters['fail']} skip={counters['skip']}",
|
||||
flush=True,
|
||||
)
|
||||
return 0 if counters["fail"] == 0 else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(asyncio.run(main()))
|
||||
|
|
@ -1,202 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Summarise the ``agentic_answer`` results across all LongMemEval samples.
|
||||
|
||||
Reports progress (how many of the 500 samples produced ``mem_answer.json``) and a
|
||||
breakdown of answer *status*:
|
||||
- answered — a non-empty answer that is not "not provided";
|
||||
- not_provided — the agent gave up ("not provided");
|
||||
- empty — ``mem_answer.json`` exists but the answer is blank;
|
||||
- missing — no ``mem_answer.json`` yet.
|
||||
|
||||
Everything is broken down by ``question_type``. This script does NOT judge answer
|
||||
correctness (there is no grader for ``mem_answer`` yet) — it only tracks progress
|
||||
and collects predicted-vs-golden pairs. Tool-call statistics are read from the
|
||||
aggregate written by ``run_agentic_answer.py`` when it is present.
|
||||
|
||||
Examples:
|
||||
python benchmark/longmemeval/stats_agentic_answer.py
|
||||
python benchmark/longmemeval/stats_agentic_answer.py --list-run-failed
|
||||
python benchmark/longmemeval/stats_agentic_answer.py --list-unanswered
|
||||
python benchmark/longmemeval/stats_agentic_answer.py --json
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
DATA = REPO / "datasets" / "longmemeval"
|
||||
LOGBASE = REPO / "logs" / "agentic_answer"
|
||||
AGGREGATE = LOGBASE / "aggregate.json"
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
"""Parse command-line arguments."""
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--list-unanswered", action="store_true", help="list samples answered 'not provided' or empty")
|
||||
p.add_argument("--list-run-failed", action="store_true", help="list launched samples with no readable output")
|
||||
p.add_argument("--json", action="store_true", help="emit the summary as JSON")
|
||||
return p.parse_args()
|
||||
|
||||
|
||||
def sample_ids() -> list[str]:
|
||||
"""List all sample IDs (numeric workspace dirs), numerically sorted."""
|
||||
ids = [p.name for p in DATA.iterdir() if p.is_dir() and p.name.isdigit()]
|
||||
return sorted(ids, key=int)
|
||||
|
||||
|
||||
def pct(num: int, den: int) -> str:
|
||||
"""Format a percentage."""
|
||||
return f"{(100.0 * num / den):.1f}%" if den else "n/a"
|
||||
|
||||
|
||||
def logged_sample_ids() -> list[str]:
|
||||
"""List sample IDs that have an agentic_answer launch log."""
|
||||
logdir = LOGBASE / "agentic_answer"
|
||||
if not logdir.exists():
|
||||
return []
|
||||
ids = [p.stem for p in logdir.glob("*.log") if p.stem.isdigit()]
|
||||
return sorted(ids, key=int)
|
||||
|
||||
|
||||
def answer_status(pred: str, has_file: bool) -> str:
|
||||
"""Classify an answer into answered / not_provided / empty / missing."""
|
||||
if not has_file:
|
||||
return "missing"
|
||||
if not pred:
|
||||
return "empty"
|
||||
if "not provided" in pred.lower():
|
||||
return "not_provided"
|
||||
return "answered"
|
||||
|
||||
|
||||
def load_tool_calls() -> dict[str, int]:
|
||||
"""Map idx -> num_tool_calls from the aggregate, if it exists."""
|
||||
if not AGGREGATE.exists():
|
||||
return {}
|
||||
try:
|
||||
with AGGREGATE.open(encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return {}
|
||||
return {s["idx"]: s.get("num_tool_calls", 0) for s in data.get("samples", []) if "idx" in s}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Main entry point."""
|
||||
args = parse_args()
|
||||
ids = sample_ids()
|
||||
total = len(ids)
|
||||
tool_calls = load_tool_calls()
|
||||
|
||||
rows, unreadable = [], []
|
||||
finished_ids = set()
|
||||
for idx in ids:
|
||||
query_path = DATA / idx / "query.json"
|
||||
mem_path = DATA / idx / "mem_answer.json"
|
||||
qtype = "(unknown)"
|
||||
try:
|
||||
with query_path.open(encoding="utf-8") as f:
|
||||
qtype = json.load(f).get("question_type") or "(unknown)"
|
||||
except (OSError, json.JSONDecodeError):
|
||||
pass
|
||||
|
||||
has_file = mem_path.exists()
|
||||
pred = ""
|
||||
if has_file:
|
||||
try:
|
||||
with mem_path.open(encoding="utf-8") as f:
|
||||
pred = str(json.load(f).get("answer") or "").strip()
|
||||
finished_ids.add(idx)
|
||||
except (OSError, json.JSONDecodeError):
|
||||
unreadable.append(idx)
|
||||
has_file = False
|
||||
|
||||
rows.append({"idx": idx, "type": qtype, "status": answer_status(pred, has_file)})
|
||||
|
||||
finished = [r for r in rows if r["status"] != "missing"]
|
||||
n = len(finished)
|
||||
launched = logged_sample_ids()
|
||||
run_failed = [idx for idx in launched if idx not in finished_ids]
|
||||
|
||||
# Overall status tallies.
|
||||
status_counts: dict[str, int] = defaultdict(int)
|
||||
for r in rows:
|
||||
status_counts[r["status"]] += 1
|
||||
answered = status_counts["answered"]
|
||||
unanswered = [r["idx"] for r in rows if r["status"] in ("not_provided", "empty")]
|
||||
|
||||
calls_vals = [tool_calls[i] for i in finished_ids if i in tool_calls]
|
||||
avg_calls = sum(calls_vals) / len(calls_vals) if calls_vals else 0.0
|
||||
|
||||
# Per question_type breakdown.
|
||||
by_type: dict[str, dict[str, int]] = defaultdict(lambda: {"n": 0, "answered": 0})
|
||||
for r in finished:
|
||||
by_type[r["type"]]["n"] += 1
|
||||
by_type[r["type"]]["answered"] += 1 if r["status"] == "answered" else 0
|
||||
|
||||
if args.json:
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"total": total,
|
||||
"finished": n,
|
||||
"pending": total - n - len(unreadable),
|
||||
"unreadable": unreadable,
|
||||
"launched": len(launched),
|
||||
"run_failed": run_failed,
|
||||
"status_counts": dict(status_counts),
|
||||
"answered_rate": round(answered / n, 4) if n else None,
|
||||
"avg_tool_calls": round(avg_calls, 2) if calls_vals else None,
|
||||
"by_type": {
|
||||
t: {**c, "answered_rate": round(c["answered"] / c["n"], 4)} for t, c in by_type.items()
|
||||
},
|
||||
"unanswered": unanswered,
|
||||
"aggregate": str(AGGREGATE) if AGGREGATE.exists() else None,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
),
|
||||
)
|
||||
return 0
|
||||
|
||||
print("=" * 60)
|
||||
print("LongMemEval agentic_answer 统计")
|
||||
print("=" * 60)
|
||||
print(f"样例总数 : {total}")
|
||||
print(f"已完成 (有产出) : {n} ({pct(n, total)})")
|
||||
print(f"未完成 : {total - n - len(unreadable)}")
|
||||
if unreadable:
|
||||
print(f"损坏/无法解析 : {len(unreadable)} {unreadable}")
|
||||
print(f"已启动过 (有 log) : {len(launched)}")
|
||||
print(f"运行失败/无可读产出 : {len(run_failed)}")
|
||||
print("-" * 60)
|
||||
print(f"已作答 (非 not provided): {answered} ({pct(answered, n)} of finished)")
|
||||
print(f" 其中 not provided : {status_counts['not_provided']}")
|
||||
print(f" 其中 空答案 : {status_counts['empty']}")
|
||||
if calls_vals:
|
||||
print(f"平均工具调用次数 : {avg_calls:.1f} (来自 {AGGREGATE.name})")
|
||||
else:
|
||||
print("平均工具调用次数 : n/a (先跑 run_agentic_answer.py 生成 aggregate.json)")
|
||||
print("-" * 60)
|
||||
print("按 question_type:")
|
||||
print(f" {'type':<24} {'n':>4} {'已作答率':>12}")
|
||||
for t in sorted(by_type):
|
||||
c = by_type[t]
|
||||
print(f" {t:<24} {c['n']:>4} {pct(c['answered'], c['n']):>12}")
|
||||
|
||||
if args.list_unanswered:
|
||||
print("-" * 60)
|
||||
print(f"not provided / 空答案的样例 ({len(unanswered)}): {unanswered}")
|
||||
if args.list_run_failed:
|
||||
print("-" * 60)
|
||||
print(f"运行失败/无可读 mem_answer.json 的样例 ({len(run_failed)}): {run_failed}")
|
||||
for idx in run_failed:
|
||||
print(f" {idx}: {LOGBASE / 'agentic_answer' / f'{idx}.log'}")
|
||||
print("=" * 60)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
|
@ -1,344 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Summarise the ``check_golden.json`` verdicts across all LongMemEval samples.
|
||||
|
||||
Reports progress (how many of the 500 samples have finished) and accuracy:
|
||||
- golden answer accuracy = share of finished samples whose golden answer the
|
||||
auditor judged correct (``verdict.golden_answer_correct``);
|
||||
- answer_session_ids accuracy = share whose claimed answer sessions the auditor
|
||||
judged exactly correct (``verdict.answer_session_ids_correct``).
|
||||
|
||||
Everything is also broken down by ``question_type``. Use ``--list-bad`` to print
|
||||
the samples whose golden answer was judged NOT correct.
|
||||
|
||||
Examples:
|
||||
python benchmark/longmemeval/stats_golden_check.py
|
||||
python benchmark/longmemeval/stats_golden_check.py --list-bad
|
||||
python benchmark/longmemeval/stats_golden_check.py --list-run-failed
|
||||
python benchmark/longmemeval/stats_golden_check.py --json
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
DATA = REPO / "datasets" / "longmemeval"
|
||||
LOGDIR = REPO / "logs" / "golden_check"
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
"""Parse command-line arguments."""
|
||||
p = argparse.ArgumentParser(description=__doc__)
|
||||
p.add_argument("--list-bad", action="store_true", help="list samples whose golden answer is NOT correct")
|
||||
p.add_argument(
|
||||
"--list-bad-sessions",
|
||||
action="store_true",
|
||||
help="list samples whose answer_session_ids is NOT correct",
|
||||
)
|
||||
p.add_argument(
|
||||
"--list-run-failed",
|
||||
action="store_true",
|
||||
help="list launched samples that did not produce readable output",
|
||||
)
|
||||
p.add_argument("--json", action="store_true", help="emit the summary as JSON")
|
||||
return p.parse_args()
|
||||
|
||||
|
||||
def sample_ids() -> list[str]:
|
||||
"""List all sample IDs."""
|
||||
ids = [p.name for p in DATA.iterdir() if p.is_dir() and p.name.isdigit()]
|
||||
return sorted(ids, key=int)
|
||||
|
||||
|
||||
def pct(num: int, den: int) -> str:
|
||||
"""Format a percentage."""
|
||||
return f"{(100.0 * num / den):.1f}%" if den else "n/a"
|
||||
|
||||
|
||||
def logged_sample_ids() -> list[str]:
|
||||
"""List all sample IDs that have been launched but not finished."""
|
||||
if not LOGDIR.exists():
|
||||
return []
|
||||
ids = [p.stem for p in LOGDIR.glob("*.log") if p.stem.isdigit()]
|
||||
return sorted(ids, key=int)
|
||||
|
||||
|
||||
def load_json(path: Path) -> dict:
|
||||
"""Load a JSON object, returning {} on any error."""
|
||||
try:
|
||||
with path.open(encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
return data if isinstance(data, dict) else {}
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return {}
|
||||
|
||||
|
||||
def question_type_for(idx: str, data: dict) -> str:
|
||||
"""Return question_type from the output, session review, or query.json."""
|
||||
question_type = str(data.get("question_type") or "").strip()
|
||||
if question_type:
|
||||
return question_type
|
||||
|
||||
review_path_raw = str(data.get("session_review_path") or "").strip()
|
||||
review_path = Path(review_path_raw) if review_path_raw else DATA / idx / "session_review.json"
|
||||
if not review_path.is_absolute():
|
||||
review_path = REPO / review_path
|
||||
review = load_json(review_path)
|
||||
review_question_type = str((review.get("query") or {}).get("question_type") or "").strip()
|
||||
if review_question_type:
|
||||
return review_question_type
|
||||
|
||||
query = load_json(DATA / idx / "query.json")
|
||||
return str(query.get("question_type") or "(unknown)").strip() or "(unknown)"
|
||||
|
||||
|
||||
def question_id_for(idx: str, data: dict) -> str:
|
||||
"""Return question_id from the output, session review, or query.json."""
|
||||
question_id = str(data.get("question_id") or "").strip()
|
||||
if question_id:
|
||||
return question_id
|
||||
|
||||
review_path_raw = str(data.get("session_review_path") or "").strip()
|
||||
review_path = Path(review_path_raw) if review_path_raw else DATA / idx / "session_review.json"
|
||||
if not review_path.is_absolute():
|
||||
review_path = REPO / review_path
|
||||
review = load_json(review_path)
|
||||
review_question_id = str((review.get("query") or {}).get("question_id") or "").strip()
|
||||
if review_question_id:
|
||||
return review_question_id
|
||||
|
||||
query = load_json(DATA / idx / "query.json")
|
||||
return str(query.get("question_id") or "").strip()
|
||||
|
||||
|
||||
def sample_label(data: dict) -> str:
|
||||
"""Format sample id as idx(question_id) when question_id is available."""
|
||||
idx = str(data.get("_idx") or "")
|
||||
qid = str(data.get("_question_id") or "").strip()
|
||||
return f"{idx}({qid})" if qid else idx
|
||||
|
||||
|
||||
def related_session_ids(data: dict) -> list[str]:
|
||||
"""Return the best available session ids for a bad verdict record."""
|
||||
verdict = data.get("verdict") if isinstance(data, dict) else None
|
||||
if isinstance(verdict, dict):
|
||||
true_ids = verdict.get("true_answer_session_ids")
|
||||
if isinstance(true_ids, list):
|
||||
ids = [str(session_id) for session_id in true_ids if str(session_id).strip()]
|
||||
if ids:
|
||||
return ids
|
||||
|
||||
summaries = data.get("session_summaries")
|
||||
if isinstance(summaries, list):
|
||||
return [
|
||||
str(summary.get("session_id"))
|
||||
for summary in summaries
|
||||
if isinstance(summary, dict) and str(summary.get("session_id") or "").strip()
|
||||
]
|
||||
return []
|
||||
|
||||
|
||||
def grouped_records(records: list[dict]) -> dict[str, list[dict]]:
|
||||
"""Group records by question_type for human-readable list output."""
|
||||
grouped: dict[str, list[dict]] = defaultdict(list)
|
||||
for data in records:
|
||||
question_type = str(data.get("_question_type") or "(unknown)")
|
||||
grouped[question_type].append(
|
||||
{
|
||||
"index": str(data.get("_idx") or ""),
|
||||
"question_id": str(data.get("_question_id") or ""),
|
||||
"session_id": related_session_ids(data),
|
||||
},
|
||||
)
|
||||
return dict(sorted(grouped.items()))
|
||||
|
||||
|
||||
def verdict_bool(verdict: dict, new_key: str, old_key: str) -> bool:
|
||||
"""Read a verdict boolean, accepting the old field name for compatibility."""
|
||||
if verdict.get(new_key) is True:
|
||||
return True
|
||||
if verdict.get(new_key) is False:
|
||||
return False
|
||||
return verdict.get(old_key) is True
|
||||
|
||||
|
||||
def has_current_verdict(data: dict) -> bool:
|
||||
"""Return True when ``check_golden.json`` uses the current golden_check schema."""
|
||||
verdict = data.get("verdict") if isinstance(data, dict) else None
|
||||
if not isinstance(verdict, dict):
|
||||
return False
|
||||
return isinstance(verdict.get("golden_answer_correct"), bool) and isinstance(
|
||||
verdict.get("answer_session_ids_correct"),
|
||||
bool,
|
||||
)
|
||||
|
||||
|
||||
def write_golden_check_list(done: list[dict], output_path: Path) -> None:
|
||||
"""Write all readable check_golden records as JSONL."""
|
||||
with output_path.open("w", encoding="utf-8") as f:
|
||||
for data in done:
|
||||
f.write(json.dumps(data, ensure_ascii=False))
|
||||
f.write("\n")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Main entry point."""
|
||||
args = parse_args()
|
||||
ids = sample_ids()
|
||||
total = len(ids)
|
||||
|
||||
done, unreadable, stale = [], [], []
|
||||
finished_ids = set()
|
||||
for idx in ids:
|
||||
path = DATA / idx / "check_golden.json"
|
||||
if not path.exists():
|
||||
continue
|
||||
try:
|
||||
with path.open(encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
if not has_current_verdict(data):
|
||||
stale.append(idx)
|
||||
continue
|
||||
data["_idx"] = idx
|
||||
data["_question_type"] = question_type_for(idx, data)
|
||||
data["_question_id"] = question_id_for(idx, data)
|
||||
done.append(data)
|
||||
finished_ids.add(idx)
|
||||
except (OSError, json.JSONDecodeError):
|
||||
unreadable.append(idx)
|
||||
|
||||
n = len(done)
|
||||
output_path = Path.cwd() / "golden_check_list.jsonl"
|
||||
write_golden_check_list(done, output_path)
|
||||
launched = logged_sample_ids()
|
||||
run_failed = [idx for idx in launched if idx not in finished_ids]
|
||||
|
||||
# Overall tallies.
|
||||
golden_ok = sum(
|
||||
1 for d in done if verdict_bool(d.get("verdict", {}), "golden_answer_correct", "golden_answer_reasonable")
|
||||
)
|
||||
sess_ok = sum(
|
||||
1
|
||||
for d in done
|
||||
if verdict_bool(d.get("verdict", {}), "answer_session_ids_correct", "answer_session_ids_reasonable")
|
||||
)
|
||||
both_ok = sum(
|
||||
1
|
||||
for d in done
|
||||
if verdict_bool(d.get("verdict", {}), "golden_answer_correct", "golden_answer_reasonable")
|
||||
and verdict_bool(d.get("verdict", {}), "answer_session_ids_correct", "answer_session_ids_reasonable")
|
||||
)
|
||||
|
||||
# Per question_type breakdown.
|
||||
by_type: dict[str, dict[str, int]] = defaultdict(lambda: {"n": 0, "golden_ok": 0, "sess_ok": 0, "both_ok": 0})
|
||||
for d in done:
|
||||
v = d.get("verdict", {})
|
||||
golden_is_ok = verdict_bool(v, "golden_answer_correct", "golden_answer_reasonable")
|
||||
sess_is_ok = verdict_bool(v, "answer_session_ids_correct", "answer_session_ids_reasonable")
|
||||
t = d.get("_question_type") or "(unknown)"
|
||||
by_type[t]["n"] += 1
|
||||
by_type[t]["golden_ok"] += 1 if golden_is_ok else 0
|
||||
by_type[t]["sess_ok"] += 1 if sess_is_ok else 0
|
||||
by_type[t]["both_ok"] += 1 if golden_is_ok and sess_is_ok else 0
|
||||
|
||||
bad_golden_records = [
|
||||
d for d in done if not verdict_bool(d.get("verdict", {}), "golden_answer_correct", "golden_answer_reasonable")
|
||||
]
|
||||
bad_session_records = [
|
||||
d
|
||||
for d in done
|
||||
if not verdict_bool(d.get("verdict", {}), "answer_session_ids_correct", "answer_session_ids_reasonable")
|
||||
]
|
||||
bad_golden = [d["_idx"] for d in bad_golden_records]
|
||||
bad_sessions = [d["_idx"] for d in bad_session_records]
|
||||
|
||||
if args.json:
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"total": total,
|
||||
"finished": n,
|
||||
"pending": total - n - len(unreadable),
|
||||
"unreadable": unreadable,
|
||||
"stale": stale,
|
||||
"launched": len(launched),
|
||||
"run_failed": run_failed,
|
||||
"golden_answer_accuracy": round(golden_ok / n, 4) if n else None,
|
||||
"answer_session_ids_accuracy": round(sess_ok / n, 4) if n else None,
|
||||
"both_correct_rate": round(both_ok / n, 4) if n else None,
|
||||
"golden_ok": golden_ok,
|
||||
"sess_ok": sess_ok,
|
||||
"both_ok": both_ok,
|
||||
"by_type": {
|
||||
t: {
|
||||
**c,
|
||||
"golden_bad": c["n"] - c["golden_ok"],
|
||||
"session_bad": c["n"] - c["sess_ok"],
|
||||
"both_bad": c["n"] - c["both_ok"],
|
||||
"golden_acc": round(c["golden_ok"] / c["n"], 4),
|
||||
"session_acc": round(c["sess_ok"] / c["n"], 4),
|
||||
"both_acc": round(c["both_ok"] / c["n"], 4),
|
||||
}
|
||||
for t, c in by_type.items()
|
||||
},
|
||||
"bad_golden": bad_golden,
|
||||
"bad_sessions": bad_sessions,
|
||||
"golden_check_list": str(output_path),
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
),
|
||||
)
|
||||
return 0
|
||||
|
||||
print("=" * 60)
|
||||
print("LongMemEval golden_check 统计")
|
||||
print("=" * 60)
|
||||
print(f"样例总数 : {total}")
|
||||
print(f"已完成 (有产出) : {n} ({pct(n, total)})")
|
||||
print(f"未完成 : {total - n - len(unreadable)}")
|
||||
if unreadable:
|
||||
print(f"损坏/无法解析 : {len(unreadable)} {unreadable}")
|
||||
if stale:
|
||||
print(f"旧格式待重跑 : {len(stale)} {stale}")
|
||||
print(f"已合并 JSONL : {output_path}")
|
||||
print(f"已启动过 (有 log) : {len(launched)}")
|
||||
print(f"运行失败/无可读产出 : {len(run_failed)}")
|
||||
print("-" * 60)
|
||||
print(f"golden answer 正确率 : {pct(golden_ok, n)} ({golden_ok}/{n})")
|
||||
print(f"answer_session 正确率: {pct(sess_ok, n)} ({sess_ok}/{n})")
|
||||
print(f"两者都正确 : {pct(both_ok, n)} ({both_ok}/{n})")
|
||||
print("-" * 60)
|
||||
print("按 question_type:")
|
||||
print(
|
||||
f" {'type':<24} {'n':>4} {'golden正确率':>14} {'golden错误':>10} "
|
||||
f"{'session正确率':>14} {'session错误':>11} {'都正确':>10} {'都正确错误':>12}",
|
||||
)
|
||||
for t in sorted(by_type):
|
||||
c = by_type[t]
|
||||
print(
|
||||
f" {t:<24} {c['n']:>4} {pct(c['golden_ok'], c['n']):>14} {c['n'] - c['golden_ok']:>10} "
|
||||
f"{pct(c['sess_ok'], c['n']):>14} {c['n'] - c['sess_ok']:>11} "
|
||||
f"{pct(c['both_ok'], c['n']):>10} {c['n'] - c['both_ok']:>12}",
|
||||
)
|
||||
|
||||
if args.list_bad:
|
||||
print("-" * 60)
|
||||
print(f"golden answer 判为不正确的样例 ({len(bad_golden_records)}):")
|
||||
print(json.dumps(grouped_records(bad_golden_records), ensure_ascii=False))
|
||||
if args.list_bad_sessions:
|
||||
print("-" * 60)
|
||||
print(f"answer_session_ids 判为不正确的样例 ({len(bad_session_records)}):")
|
||||
print(json.dumps(grouped_records(bad_session_records), ensure_ascii=False))
|
||||
if args.list_run_failed:
|
||||
print("-" * 60)
|
||||
print(f"运行失败/无可读 check_golden.json 的样例 ({len(run_failed)}): {run_failed}")
|
||||
for idx in run_failed:
|
||||
print(f" {idx}: {LOGDIR / f'{idx}.log'}")
|
||||
print("=" * 60)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
|
@ -1,248 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Summarise LongMemEval ``session_review.json`` artifacts.
|
||||
|
||||
This script is for upstream health checks before running ``golden_check``.
|
||||
Samples with retryable per-session failures should be rerun as a whole; samples
|
||||
with non-retryable fallback reviews are reported separately.
|
||||
|
||||
Examples:
|
||||
python benchmark/longmemeval/stats_session_review.py
|
||||
python benchmark/longmemeval/stats_session_review.py --list-failed
|
||||
python benchmark/longmemeval/stats_session_review.py --list-fallback
|
||||
python benchmark/longmemeval/stats_session_review.py --json
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
DATA = REPO / "datasets" / "longmemeval"
|
||||
LOGDIR = REPO / "logs" / "session_review"
|
||||
OUTPUT_FILENAME = "session_review.json"
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
"""Parse command-line arguments."""
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--list-failed", action="store_true", help="list samples with retryable failed per-session reviews")
|
||||
p.add_argument("--list-fallback", action="store_true", help="list non-retryable fallback reviews")
|
||||
p.add_argument("--list-missing", action="store_true", help="list samples missing session_review.json")
|
||||
p.add_argument("--list-run-failed", action="store_true", help="list launched samples without a healthy output")
|
||||
p.add_argument("--json", action="store_true", help="emit the summary as JSON")
|
||||
return p.parse_args()
|
||||
|
||||
|
||||
def sample_ids() -> list[str]:
|
||||
"""List all numeric sample IDs."""
|
||||
ids = [p.name for p in DATA.iterdir() if p.is_dir() and p.name.isdigit()]
|
||||
return sorted(ids, key=int)
|
||||
|
||||
|
||||
def pct(num: int, den: int) -> str:
|
||||
"""Format a percentage."""
|
||||
return f"{(100.0 * num / den):.1f}%" if den else "n/a"
|
||||
|
||||
|
||||
def load_json(path: Path) -> dict:
|
||||
"""Load a JSON object, returning {} on any error."""
|
||||
try:
|
||||
with path.open(encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
return data if isinstance(data, dict) else {}
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return {}
|
||||
|
||||
|
||||
def logged_sample_ids() -> list[str]:
|
||||
"""List sample IDs that have a session_review runner log."""
|
||||
if not LOGDIR.exists():
|
||||
return []
|
||||
ids = [p.stem for p in LOGDIR.glob("*.log") if p.stem.isdigit()]
|
||||
return sorted(ids, key=int)
|
||||
|
||||
|
||||
def review_block(data: dict) -> dict:
|
||||
"""Return the review block when present."""
|
||||
review = data.get("review") if isinstance(data, dict) else None
|
||||
return review if isinstance(review, dict) else {}
|
||||
|
||||
|
||||
def failure_details(data: dict) -> list[dict]:
|
||||
"""Return retryable failed_reviews when present."""
|
||||
failed_reviews = review_block(data).get("failed_reviews")
|
||||
if not isinstance(failed_reviews, list):
|
||||
return []
|
||||
return [item for item in failed_reviews if isinstance(item, dict) and not item.get("fallback")]
|
||||
|
||||
|
||||
def fallback_details(data: dict) -> list[dict]:
|
||||
"""Return non-retryable fallback review details when present."""
|
||||
review = review_block(data)
|
||||
fallback_reviews = review.get("fallback_reviews")
|
||||
if isinstance(fallback_reviews, list):
|
||||
return [item for item in fallback_reviews if isinstance(item, dict)]
|
||||
|
||||
failed_reviews = review.get("failed_reviews")
|
||||
if isinstance(failed_reviews, list):
|
||||
return [item for item in failed_reviews if isinstance(item, dict) and item.get("fallback")]
|
||||
return []
|
||||
|
||||
|
||||
def failure_count(data: dict) -> int:
|
||||
"""Return retryable failed review count."""
|
||||
review = review_block(data)
|
||||
raw = review.get("num_failed_reviews")
|
||||
raw_fallback = review.get("num_fallback_reviews")
|
||||
if isinstance(raw, int) and isinstance(raw_fallback, int):
|
||||
return max(0, raw - raw_fallback)
|
||||
return len(failure_details(data))
|
||||
|
||||
|
||||
def fallback_count(data: dict) -> int:
|
||||
"""Return non-retryable fallback review count."""
|
||||
review = review_block(data)
|
||||
raw = review.get("num_fallback_reviews")
|
||||
if isinstance(raw, int):
|
||||
return raw
|
||||
return len(fallback_details(data))
|
||||
|
||||
|
||||
def question_id(data: dict) -> str:
|
||||
"""Return query.question_id when present."""
|
||||
query = data.get("query") if isinstance(data, dict) else None
|
||||
if not isinstance(query, dict):
|
||||
return ""
|
||||
return str(query.get("question_id") or "").strip()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Main entry point."""
|
||||
args = parse_args()
|
||||
ids = sample_ids()
|
||||
total = len(ids)
|
||||
|
||||
healthy, failed, fallback, missing, unreadable = [], [], [], [], []
|
||||
total_failed_sessions = 0
|
||||
total_fallback_sessions = 0
|
||||
failed_details_by_id: dict[str, list[dict]] = {}
|
||||
fallback_details_by_id: dict[str, list[dict]] = {}
|
||||
question_id_by_id: dict[str, str] = {}
|
||||
|
||||
for idx in ids:
|
||||
path = DATA / idx / OUTPUT_FILENAME
|
||||
if not path.exists():
|
||||
missing.append(idx)
|
||||
continue
|
||||
data = load_json(path)
|
||||
if not data:
|
||||
unreadable.append(idx)
|
||||
continue
|
||||
question_id_by_id[idx] = question_id(data)
|
||||
n_failed = failure_count(data)
|
||||
n_fallback = fallback_count(data)
|
||||
if n_failed:
|
||||
failed.append(idx)
|
||||
total_failed_sessions += n_failed
|
||||
failed_details_by_id[idx] = failure_details(data)
|
||||
if n_fallback:
|
||||
fallback.append(idx)
|
||||
total_fallback_sessions += n_fallback
|
||||
fallback_details_by_id[idx] = fallback_details(data)
|
||||
if not n_failed:
|
||||
healthy.append(idx)
|
||||
|
||||
launched = logged_sample_ids()
|
||||
healthy_set = set(healthy)
|
||||
run_failed = [idx for idx in launched if idx not in healthy_set]
|
||||
|
||||
if args.json:
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"total": total,
|
||||
"healthy": len(healthy),
|
||||
"failed_samples": failed,
|
||||
"failed_sample_count": len(failed),
|
||||
"failed_session_count": total_failed_sessions,
|
||||
"fallback_samples": fallback,
|
||||
"fallback_sample_count": len(fallback),
|
||||
"fallback_session_count": total_fallback_sessions,
|
||||
"missing": missing,
|
||||
"unreadable": unreadable,
|
||||
"launched": len(launched),
|
||||
"run_failed_or_unhealthy": run_failed,
|
||||
"failed_details": failed_details_by_id,
|
||||
"fallback_details": fallback_details_by_id,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
),
|
||||
)
|
||||
return 0
|
||||
|
||||
print("=" * 60)
|
||||
print("LongMemEval session_review 统计")
|
||||
print("=" * 60)
|
||||
print(f"样例总数 : {total}")
|
||||
print(f"可继续产出 : {len(healthy)} ({pct(len(healthy), total)})")
|
||||
print(f"有可重试失败 : {len(failed)}")
|
||||
print(f"可重试失败 session : {total_failed_sessions}")
|
||||
print(f"有不可重试 fallback : {len(fallback)}")
|
||||
print(f"fallback session : {total_fallback_sessions}")
|
||||
print(f"缺少 session_review : {len(missing)}")
|
||||
print(f"损坏/无法解析 : {len(unreadable)}")
|
||||
print(f"已启动过 (有 log) : {len(launched)}")
|
||||
print(f"运行失败/非健康产出 : {len(run_failed)}")
|
||||
print("-" * 60)
|
||||
print("有可重试 failed_reviews 的样例需要整体重跑：")
|
||||
if failed:
|
||||
print(" ".join(failed))
|
||||
print("重跑命令示例：")
|
||||
print(f"python benchmark/longmemeval/run_session_review.py --start {failed[0]} --end {failed[0]}")
|
||||
else:
|
||||
print("(none)")
|
||||
if fallback:
|
||||
print("-" * 60)
|
||||
print("不可重试 fallback 的样例不用重跑：")
|
||||
for idx in fallback:
|
||||
details = fallback_details_by_id.get(idx) or []
|
||||
session_ids = [str(item.get("session_id") or "(unknown)") for item in details]
|
||||
qid = question_id_by_id.get(idx)
|
||||
sample_label = f"{idx}({qid})" if qid else idx
|
||||
print(f"{sample_label}: {' '.join(session_ids) if session_ids else '(unknown)'}")
|
||||
|
||||
if args.list_failed and failed:
|
||||
print("-" * 60)
|
||||
for idx in failed:
|
||||
details = failed_details_by_id.get(idx) or []
|
||||
print(f"{idx}: {DATA / idx / OUTPUT_FILENAME} failed_sessions={len(details)}")
|
||||
for item in details:
|
||||
session_id = item.get("session_id", "(unknown)")
|
||||
error = str(item.get("error") or "").replace("\n", " ")
|
||||
print(f" - {session_id}: {error}")
|
||||
if args.list_fallback and fallback:
|
||||
print("-" * 60)
|
||||
for idx in fallback:
|
||||
details = fallback_details_by_id.get(idx) or []
|
||||
print(f"{idx}: {DATA / idx / OUTPUT_FILENAME} fallback_sessions={len(details)}")
|
||||
for item in details:
|
||||
session_id = item.get("session_id", "(unknown)")
|
||||
reason = str(item.get("fallback_reason") or "fallback")
|
||||
error = str(item.get("error") or "").replace("\n", " ")
|
||||
raw_saved = "yes" if item.get("raw_session") else "no"
|
||||
print(f" - {session_id}: reason={reason} raw_session_saved={raw_saved} error={error}")
|
||||
if args.list_missing and missing:
|
||||
print("-" * 60)
|
||||
print(f"缺少 session_review.json 的样例 ({len(missing)}): {missing}")
|
||||
if args.list_run_failed and run_failed:
|
||||
print("-" * 60)
|
||||
print(f"运行失败/非健康产出的样例 ({len(run_failed)}): {run_failed}")
|
||||
for idx in run_failed:
|
||||
print(f" {idx}: {LOGDIR / f'{idx}.log'}")
|
||||
print("=" * 60)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
36
benchmark/result-beam.md
Normal file
36
benchmark/result-beam.md
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
# beam result
|
||||
|
||||
## longmemeval版本的prompt
|
||||
|
||||
### 100K
|
||||
|
||||
|
||||
| 题型 | Prompted(limit=15) | Agentic |
|
||||
|---|---|---|
|
||||
| abstention | 0.525 | 0.575 |
|
||||
| contradiction_resolution | 0.100 | 0.384 |
|
||||
| event_ordering | 0.403 | 0.465 |
|
||||
| information_extraction | 0.618 | 0.884 |
|
||||
| instruction_following | 0.481 | 0.719 |
|
||||
| knowledge_update | 0.637 | 0.650 |
|
||||
| multi_session_reasoning | 0.444 | 0.633 |
|
||||
| preference_following | 0.706 | 0.829 |
|
||||
| summarization | 0.423 | 0.617 |
|
||||
| temporal_reasoning | 0.344 | 0.550 |
|
||||
| **OVERALL** | **0.468** | **0.631** |
|
||||
|
||||
### 1M
|
||||
|
||||
| 题型 | Prompted(limit=15) | Prompted Binary | Agentic | Agentic Binary |
|
||||
|---|---|---|---|---|
|
||||
| abstention | 0.464 | 0.464 | 0.514 | 0.514 |
|
||||
| contradiction_resolution | 0.079 | 0.068 | 0.373 | 0.339 |
|
||||
| event_ordering | 0.455 | 0.334 | 0.547 | 0.450 |
|
||||
| information_extraction | 0.653 | 0.589 | 0.818 | 0.764 |
|
||||
| instruction_following | 0.541 | 0.524 | 0.765 | 0.745 |
|
||||
| knowledge_update | 0.571 | 0.507 | 0.636 | 0.629 |
|
||||
| multi_session_reasoning | 0.426 | 0.324 | 0.593 | 0.540 |
|
||||
| preference_following | 0.718 | 0.676 | 0.838 | 0.824 |
|
||||
| summarization | 0.516 | 0.303 | 0.661 | 0.478 |
|
||||
| temporal_reasoning | 0.198 | 0.169 | 0.394 | 0.383 |
|
||||
| **OVERALL** | **0.462** | **0.396** | **0.614** | **0.567** |
|
||||
92
benchmark/result-longmemeval.md
Normal file
92
benchmark/result-longmemeval.md
Normal file
|
|
@ -0,0 +1,92 @@
|
|||
# LongMemEval 数据集测试结果
|
||||
|
||||
## cleaned-s
|
||||
|
||||
**basic settings**
|
||||
|
||||
1. 使用修改后的auto-memory prompt，关闭auto-dream机制
|
||||
2. reme-memory中的全部session的时间一定早于question的时间
|
||||
|
||||
**results **
|
||||
|
||||
1. Agentic answer框架回答，每次最多调用5次search
|
||||
|
||||
| Category | Total | Correct | Wrong | Accuracy |
|
||||
|---|---|---|---|---|
|
||||
| single-session-user | 70 | 66 | 4 | 94.3% |
|
||||
| single-session-assistant | 56 | 52 | 4 | 92.9% |
|
||||
| knowledge-update | 78 | 60 | 18 | 76.9% |
|
||||
| multi-session | 133 | 93 | 40 | 69.9% |
|
||||
| temporal-reasoning | 133 | 78 | 55 | 58.6% |
|
||||
| single-session-preference | 30 | 8 | 22 | 26.7% |
|
||||
| **Overall** | **500** | **357** | **143** | **71.4%** |
|
||||
|
||||
2. prompted-based amswer，每次固定使用原始query召回10个fileChunk
|
||||
|
||||
| Category | Total | Correct | Wrong | Accuracy |
|
||||
|---|---|---|---|---|
|
||||
| single-session-assistant | 56 | 56 | 0 | 100.0% |
|
||||
| single-session-user | 70 | 67 | 3 | 95.7% |
|
||||
| knowledge-update | 78 | 69 | 9 | 88.5% |
|
||||
| multi-session | 133 | 99 | 34 | 74.4% |
|
||||
| temporal-reasoning | 133 | 83 | 50 | 62.4% |
|
||||
| single-session-preference | 30 | 16 | 14 | 53.3% |
|
||||
| **Overall** | **500** | **390** | **110** | **78.0%** |
|
||||
|
||||
3. golden session。 使用与prompt-based answer相似的方法，唯一区别是，输入的chunk是longMemEval提供的golden session。
|
||||
|
||||
| Category | Total | Correct | Wrong | Accuracy |
|
||||
|---|---|---|---|---|
|
||||
| single-session-assistant | 56 | 56 | 0 | 100.0% |
|
||||
| single-session-user | 70 | 69 | 1 | 98.6% |
|
||||
| knowledge-update | 78 | 74 | 4 | 94.9% |
|
||||
| temporal-reasoning | 133 | 124 | 9 | 93.2% |
|
||||
| multi-session | 133 | 117 | 16 | 88.0% |
|
||||
| single-session-preference | 30 | 17 | 13 | 56.7% |
|
||||
| **Overall** | **500** | **457** | **43** | **91.4%** |
|
||||
|
||||
4. golden session + time filter. 和上面一个实验的区别是，输入的golden被过滤了一次，要求输入session的时间戳必须早于question的时间才行。
|
||||
|
||||
一共被过滤掉了75个session，44个question受到了影响。temperal reasoning类型受影响最大。有20个case不包含任何一个groundtruth session。 根据golden session回答正确并且golden session非空，一共有424个case。
|
||||
|
||||
| Category | Total | Correct | Wrong | Accuracy |
|
||||
|---|---|---|---|---|
|
||||
| knowledge-update | 78 | 75 | 3 | 96.2% |
|
||||
| single-session-user | 70 | 67 | 3 | 95.7% |
|
||||
| multi-session | 133 | 122 | 11 | 91.7% |
|
||||
| single-session-assistant | 56 | 55 | 1 | 98.2% |
|
||||
| temporal-reasoning | 133 | 91 | 42 | 68.4% |
|
||||
| single-session-preference | 30 | 16 | 14 | 53.3% |
|
||||
| **Overall** | **500** | **426** | **74** | **85.2%** |
|
||||
|
||||
5. 关闭auto-memory机制，根据原始query一次性混合检索召回原始session，计算recall.
|
||||
|
||||
| Category | Total | yes-judge | recall@5 / yes | recall@10 / yes |
|
||||
|---|---|---|---|---|
|
||||
| knowledge-update | 78 | 75 | 99.3% | 100% |
|
||||
| single-session-user | 70 | 67 | 100% | 100% |
|
||||
| multi-session | 133 | 122 | 91.8% | 95.8% |
|
||||
| single-session-assistant | 56 | 55 | 100% | 100% |
|
||||
| temporal-reasoning | 133 | 91 | 87.6% | 94.2% |
|
||||
| single-session-preference | 30 | 16 | 100% | 100% |
|
||||
| **Overall** | **500** | **426** | **87.6%** | **94.2%** |
|
||||
|
||||
|
||||
## 最终groundtruth
|
||||
|
||||
### agentic + prompted（最终GT，2026-07-16）
|
||||
|
||||
|
||||
| Category | Total | Agentic | Prompted limit=15 |
|
||||
|---|---|---|---|
|
||||
| single-session-assistant | 56 | 56/56 (100.0%) | 54/56 (96.4%) |
|
||||
| single-session-user | 70 | 66/70 (94.3%) | 62/70 (88.6%) |
|
||||
| knowledge-update | 78 | 75/78 (96.2%) | 67/78 (85.9%) |
|
||||
| temporal-reasoning | 133 | 122/133 (91.7%) | 117/133 (88.0%) |
|
||||
| multi-session | 133 | 115/133 (86.5%) | 101/133 (75.9%) |
|
||||
| single-session-preference | 30 | 21/30 (70.0%) | 10/30 (33.3%) |
|
||||
| **Overall** | **500** | **455/500 (91.0%)** | **411/500 (82.2%)** |
|
||||
|
||||
Prompted token 消耗：总 input 13,111,421 (平均 26,275/题)，总 output 313,370 (平均 628/题)。
|
||||
平均 sessions_ingested: 44.8，dreams_triggered: 0。
|
||||
|
||||
586
reme/config/beam.yaml
Normal file
586
reme/config/beam.yaml
Normal file
|
|
@ -0,0 +1,586 @@
|
|||
# BEAM benchmark config — based on longmemeval.yaml (split)
|
||||
# All background/cron jobs are converted to base (manually callable).
|
||||
# Use with: resolve_app_config(config="beam.yaml", ...)
|
||||
|
||||
service:
|
||||
backend: http
|
||||
|
||||
jobs:
|
||||
# ── BEAM agentic answer (ReAct agent + search tool) ──
|
||||
agentic_answer:
|
||||
backend: base
|
||||
description: "BEAM agentic answer job (ReAct agent with search tool)"
|
||||
watch_dirs: []
|
||||
watch_suffixes: []
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
query:
|
||||
type: string
|
||||
description: "The query to ask"
|
||||
query_time:
|
||||
type: string
|
||||
description: "ISO timestamp representing the query time"
|
||||
default: ""
|
||||
required:
|
||||
- query
|
||||
steps:
|
||||
- backend: beam_agentic_answer_step
|
||||
agent_wrapper: bench
|
||||
|
||||
# ── BEAM rubric-based LLM-as-Judge ──
|
||||
answer_judge:
|
||||
backend: base
|
||||
description: "BEAM rubric-based LLM-as-Judge: evaluate response against rubric criteria"
|
||||
watch_dirs: []
|
||||
watch_suffixes: []
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
llm_response:
|
||||
type: string
|
||||
description: "The model's response to evaluate"
|
||||
rubric:
|
||||
type: array
|
||||
description: "List of rubric criteria to check"
|
||||
items:
|
||||
type: string
|
||||
probing_question:
|
||||
type: string
|
||||
description: "The original probing question"
|
||||
default: ""
|
||||
question_type:
|
||||
type: string
|
||||
description: "BEAM question type (e.g. event_ordering)"
|
||||
default: ""
|
||||
required:
|
||||
- llm_response
|
||||
- rubric
|
||||
steps:
|
||||
- backend: beam_rubric_judge_step
|
||||
agent_wrapper: judge
|
||||
|
||||
# ── Manual index update (replaces index_update_loop background) ──
|
||||
index_update:
|
||||
backend: base
|
||||
description: "Manually trigger incremental index update for watched dirs."
|
||||
watch_dirs: [daily_dir, digest_dir, dialog_dir]
|
||||
watch_suffixes: [md, jsonl]
|
||||
parameters:
|
||||
type: object
|
||||
properties: {}
|
||||
steps:
|
||||
- backend: init_changes_step
|
||||
monitor_type: file_store
|
||||
monitor_name: default
|
||||
dispatch_steps: [update_index_step]
|
||||
|
||||
# ── Manual digest catalog update (replaces digest_watch_loop background) ──
|
||||
digest_update:
|
||||
backend: base
|
||||
description: "Manually trigger digest catalog update."
|
||||
watch_dirs: [daily_dir, digest_dir]
|
||||
watch_suffixes: [md]
|
||||
parameters:
|
||||
type: object
|
||||
properties: {}
|
||||
steps:
|
||||
- backend: init_changes_step
|
||||
monitor_type: file_catalog
|
||||
monitor_name: digest
|
||||
dispatch_steps:
|
||||
- backend: update_catalog_step
|
||||
file_catalog: digest
|
||||
- backend: log_changes_step
|
||||
|
||||
# ── Auto dream (same as default.yaml auto_dream, base mode) ──
|
||||
# auto_dream:
|
||||
# backend: base
|
||||
# description: "Auto-dream: scan today's day-index and daily notes, globally extract merged units/topics, integrate digest units, write interests.yaml, and persist the dream catalog."
|
||||
# parameters:
|
||||
# type: object
|
||||
# properties:
|
||||
# date:
|
||||
# type: string
|
||||
# description: "YYYY-MM-DD to scan; defaults to today in the dreamer's timezone"
|
||||
# default: ""
|
||||
# hint:
|
||||
# type: string
|
||||
# description: "caller guidance passed through to dream extract/integrate"
|
||||
# default: ""
|
||||
# scan_days:
|
||||
# type: integer
|
||||
# description: "number of recent daily directories to scan, ending at date"
|
||||
# default: 2
|
||||
# max_units:
|
||||
# type: integer
|
||||
# description: "maximum number of extracted memory units"
|
||||
# default: 5
|
||||
# topic_count:
|
||||
# type: integer
|
||||
# description: "maximum number of final daily interest topics"
|
||||
# default: 3
|
||||
# topic_diversity_days:
|
||||
# type: integer
|
||||
# description: "number of previous interests.yaml days to avoid repeating"
|
||||
# default: 7
|
||||
# steps:
|
||||
# - backend: dream_extract_step
|
||||
# file_catalog: dream
|
||||
# topic_session_id: interests
|
||||
# scan_days: 2
|
||||
# max_units: 5
|
||||
# - backend: dream_integrate_step
|
||||
# - backend: dream_topics_step
|
||||
# topic_count: 3
|
||||
# topic_diversity_days: 7
|
||||
# - backend: dream_finish_step
|
||||
# file_catalog: dream
|
||||
|
||||
# ── Auto memory (same as default.yaml) ──
|
||||
auto_memory:
|
||||
backend: base
|
||||
description: "Auto-memory: record conversation facts into a daily note"
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
messages:
|
||||
type: array
|
||||
description: "messages"
|
||||
items:
|
||||
type: object
|
||||
session_id:
|
||||
type: string
|
||||
description: "source conversation session identifier"
|
||||
default: ""
|
||||
memory_hint:
|
||||
type: string
|
||||
description: "optional hint"
|
||||
date:
|
||||
type: string
|
||||
description: "YYYY-MM-DD daily note date; empty = infer from message timestamps or today"
|
||||
default: ""
|
||||
required:
|
||||
- messages
|
||||
steps:
|
||||
- backend: beam_auto_memory_step
|
||||
|
||||
# ── Reindex (full rebuild) ──
|
||||
reindex:
|
||||
backend: base
|
||||
description: "wipe the file store and rebuild it from the existing files"
|
||||
watch_dirs: [daily_dir, digest_dir, resource_dir]
|
||||
watch_suffixes: [md, jsonl]
|
||||
parameters:
|
||||
type: object
|
||||
properties: {}
|
||||
steps:
|
||||
- backend: clear_store_step
|
||||
- backend: init_changes_step
|
||||
monitor_type: file_store
|
||||
monitor_name: default
|
||||
dispatch_steps: [update_index_step]
|
||||
|
||||
# ── Search ──
|
||||
# start_date:
|
||||
# type: string
|
||||
# description: "optional inclusive start date filter (YYYY-MM-DD); results earlier than this date are excluded"
|
||||
# end_date:
|
||||
# type: string
|
||||
# description: "optional inclusive end date filter (YYYY-MM-DD); results later than this date are excluded"
|
||||
|
||||
search:
|
||||
backend: base
|
||||
description: "Hybrid workspace search (vector + BM25, RRF-fused) with deduplication."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
query:
|
||||
type: string
|
||||
description: "search query"
|
||||
start_date:
|
||||
type: string
|
||||
description: "optional inclusive start date filter (YYYY-MM-DD); results earlier than this date are excluded"
|
||||
end_date:
|
||||
type: string
|
||||
description: "optional inclusive end date filter (YYYY-MM-DD); results later than this date are excluded"
|
||||
# vector_weight:
|
||||
# type: number
|
||||
# description: >-
|
||||
# Optional weight balancing vector similarity vs BM25 keyword matching in the
|
||||
# RRF fusion. Recommended value is 0.7, which provides a good balance between
|
||||
# semantic (vector) similarity and lexical (BM25) matching. Values close to 0
|
||||
# emphasize BM25 keyword matching, values close to 1 emphasize vector semantic
|
||||
# similarity.
|
||||
required:
|
||||
- query
|
||||
steps:
|
||||
- backend: search_step
|
||||
vector_weight: 0.7
|
||||
candidate_multiplier: 5.0
|
||||
expand_links: true
|
||||
max_links_per_direction: 10
|
||||
|
||||
add_draft:
|
||||
backend: base
|
||||
description: "Append text to the current draft list."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
text:
|
||||
type: string
|
||||
description: "draft text to append"
|
||||
required:
|
||||
- text
|
||||
steps:
|
||||
- backend: add_draft_step
|
||||
|
||||
read_all_draft:
|
||||
backend: base
|
||||
description: "Read all draft text previously appended in the current tool context."
|
||||
parameters:
|
||||
type: object
|
||||
properties: { }
|
||||
steps:
|
||||
- backend: read_all_draft_step
|
||||
|
||||
python_execute:
|
||||
backend: base
|
||||
description: "Execute Python code and return printed stdout."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
code:
|
||||
type: string
|
||||
description: "Python code to execute. Print the final result to stdout."
|
||||
timeout:
|
||||
type: number
|
||||
description: "Execution timeout in seconds; defaults to 60."
|
||||
required:
|
||||
- code
|
||||
steps:
|
||||
- backend: python_execute_step
|
||||
|
||||
# ── File I/O jobs (needed by auto_memory agent tools) ──
|
||||
daily_list:
|
||||
backend: base
|
||||
description: "List notes under a single day."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
date:
|
||||
type: string
|
||||
description: "YYYY-MM-DD; empty = today"
|
||||
default: ""
|
||||
steps:
|
||||
- backend: daily_list_step
|
||||
|
||||
daily_reindex:
|
||||
backend: base
|
||||
description: "Rebuild the day-index page daily/<date>.md."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
date:
|
||||
type: string
|
||||
description: "YYYY-MM-DD; empty = today"
|
||||
default: ""
|
||||
steps:
|
||||
- backend: daily_reindex_step
|
||||
|
||||
frontmatter_update:
|
||||
backend: base
|
||||
description: "Merge key-values into a file's frontmatter."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
path:
|
||||
type: string
|
||||
description: "workspace-relative path"
|
||||
metadata:
|
||||
type: object
|
||||
description: "key-values to merge"
|
||||
required:
|
||||
- path
|
||||
- metadata
|
||||
steps:
|
||||
- backend: frontmatter_update_step
|
||||
|
||||
move:
|
||||
backend: base
|
||||
description: "Move / rename a workspace file."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
src_path:
|
||||
type: string
|
||||
description: "workspace-relative source"
|
||||
dst_path:
|
||||
type: string
|
||||
description: "workspace-relative destination"
|
||||
overwrite:
|
||||
type: boolean
|
||||
default: false
|
||||
retarget:
|
||||
type: boolean
|
||||
default: true
|
||||
required:
|
||||
- src_path
|
||||
- dst_path
|
||||
steps:
|
||||
- backend: move_step
|
||||
|
||||
read:
|
||||
backend: base
|
||||
description: "Read a markdown file under the workspace."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
path:
|
||||
type: string
|
||||
description: "workspace-relative path; markdown only"
|
||||
start_line:
|
||||
type: integer
|
||||
end_line:
|
||||
type: integer
|
||||
required:
|
||||
- path
|
||||
steps:
|
||||
- backend: read_step
|
||||
with_neighbors: false
|
||||
max_neighbors_per_direction: 10
|
||||
|
||||
write:
|
||||
backend: base
|
||||
description: "Write a markdown file."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
path:
|
||||
type: string
|
||||
name:
|
||||
type: string
|
||||
description:
|
||||
type: string
|
||||
content:
|
||||
type: string
|
||||
metadata:
|
||||
type: object
|
||||
required:
|
||||
- path
|
||||
- name
|
||||
- description
|
||||
- content
|
||||
steps:
|
||||
- backend: write_step
|
||||
|
||||
daily_write:
|
||||
backend: base
|
||||
description: "Write a daily markdown note."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
name:
|
||||
type: string
|
||||
description:
|
||||
type: string
|
||||
session_id:
|
||||
type: string
|
||||
content:
|
||||
type: string
|
||||
date:
|
||||
type: string
|
||||
default: ""
|
||||
metadata:
|
||||
type: object
|
||||
required:
|
||||
- name
|
||||
- description
|
||||
- session_id
|
||||
- content
|
||||
steps:
|
||||
- backend: daily_write_step
|
||||
|
||||
edit:
|
||||
backend: base
|
||||
description: "Find-and-replace in a markdown file."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
path:
|
||||
type: string
|
||||
old:
|
||||
type: string
|
||||
new:
|
||||
type: string
|
||||
default: ""
|
||||
required:
|
||||
- path
|
||||
- old
|
||||
- new
|
||||
steps:
|
||||
- backend: edit_step
|
||||
|
||||
frontmatter_read:
|
||||
backend: base
|
||||
description: "Read a file's frontmatter as a dict."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
path:
|
||||
type: string
|
||||
required:
|
||||
- path
|
||||
steps:
|
||||
- backend: frontmatter_read_step
|
||||
|
||||
node_search:
|
||||
backend: base
|
||||
description: "Digest node recall."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
query:
|
||||
type: string
|
||||
limit:
|
||||
type: integer
|
||||
default: 20
|
||||
required:
|
||||
- query
|
||||
steps:
|
||||
- backend: node_search_step
|
||||
vector_weight: 0.7
|
||||
candidate_multiplier: 5.0
|
||||
|
||||
components:
|
||||
tokenizer:
|
||||
default:
|
||||
backend: regex
|
||||
|
||||
as_embedding:
|
||||
default:
|
||||
backend: ${EMBEDDING_BACKEND:-openai}
|
||||
model: ${EMBEDDING_MODEL_NAME:-text-embedding-v4}
|
||||
credential:
|
||||
api_key: ${EMBEDDING_API_KEY:-}
|
||||
base_url: ${EMBEDDING_BASE_URL:-https://dashscope.aliyuncs.com/compatible-mode/v1}
|
||||
dimensions: 1024
|
||||
|
||||
embedding_store:
|
||||
default:
|
||||
backend: local
|
||||
as_embedding: default
|
||||
|
||||
as_llm:
|
||||
default:
|
||||
backend: ${LLM_BACKEND:-openai}
|
||||
model: ${LLM_MODEL_NAME:-qwen3.6-flash}
|
||||
stream: true
|
||||
context_size: 200000
|
||||
max_retries: 5
|
||||
retry_delay: 5.0
|
||||
credential:
|
||||
api_key: ${LLM_API_KEY:-}
|
||||
base_url: ${LLM_BASE_URL:-}
|
||||
parameters:
|
||||
max_tokens: 65536
|
||||
thinking_enable: false
|
||||
judge:
|
||||
backend: ${LLM_BACKEND:-openai}
|
||||
model: ${JUDGE_MODEL_NAME:-qwen3.7-max}
|
||||
stream: false
|
||||
context_size: 200000
|
||||
max_retries: 5
|
||||
retry_delay: 5.0
|
||||
credential:
|
||||
api_key: ${LLM_API_KEY:-}
|
||||
base_url: ${LLM_BASE_URL:-}
|
||||
parameters:
|
||||
max_tokens: 65536
|
||||
thinking_enable: false
|
||||
bench:
|
||||
backend: ${LLM_BACKEND:-openai}
|
||||
model: ${BENCH_MODEL_NAME:-qwen3.7-max}
|
||||
stream: true
|
||||
context_size: 400000
|
||||
max_retries: 5
|
||||
retry_delay: 5.0
|
||||
credential:
|
||||
api_key: ${LLM_API_KEY:-}
|
||||
base_url: ${LLM_BASE_URL:-}
|
||||
parameters:
|
||||
max_tokens: 65536
|
||||
thinking_enable: true
|
||||
|
||||
agent_wrapper:
|
||||
default:
|
||||
backend: agentscope
|
||||
as_llm: default
|
||||
permission_mode: bypass
|
||||
react_config:
|
||||
max_iters: 30
|
||||
context_config:
|
||||
trigger_ratio: 0.8
|
||||
reserve_ratio: 0.1
|
||||
tool_result_limit: 50000
|
||||
model_config:
|
||||
max_retries: 1
|
||||
judge:
|
||||
backend: agentscope
|
||||
as_llm: judge
|
||||
permission_mode: bypass
|
||||
react_config:
|
||||
max_iters: 1
|
||||
context_config:
|
||||
trigger_ratio: 0.8
|
||||
reserve_ratio: 0.1
|
||||
tool_result_limit: 50000
|
||||
model_config:
|
||||
max_retries: 1
|
||||
bench:
|
||||
backend: agentscope
|
||||
as_llm: bench
|
||||
permission_mode: bypass
|
||||
react_config:
|
||||
max_iters: 30
|
||||
context_config:
|
||||
trigger_ratio: 0.8
|
||||
reserve_ratio: 0.1
|
||||
tool_result_limit: 50000
|
||||
model_config:
|
||||
max_retries: 1
|
||||
|
||||
file_graph:
|
||||
default:
|
||||
backend: local
|
||||
|
||||
file_catalog:
|
||||
default:
|
||||
backend: local
|
||||
resource:
|
||||
backend: local
|
||||
digest:
|
||||
backend: local
|
||||
dream:
|
||||
backend: local
|
||||
|
||||
file_chunker:
|
||||
markdown:
|
||||
backend: markdown
|
||||
supported_extensions: ["md"]
|
||||
default:
|
||||
backend: default
|
||||
supported_extensions: ["jsonl"]
|
||||
|
||||
keyword_index:
|
||||
default:
|
||||
backend: bm25
|
||||
tokenizer: default
|
||||
|
||||
file_store:
|
||||
default:
|
||||
backend: local
|
||||
store_name: local
|
||||
embedding_store: default
|
||||
keyword_index: default
|
||||
file_graph: default
|
||||
|
|
@ -1,419 +0,0 @@
|
|||
service:
|
||||
backend: cli
|
||||
|
||||
workspace_dir: ${LME_WORKSPACE_DIR:-datasets/longmemeval/1}
|
||||
session_dir: history_session
|
||||
resource_dir: session
|
||||
daily_dir: daily
|
||||
digest_dir: ""
|
||||
|
||||
jobs:
|
||||
auto_memory:
|
||||
backend: base
|
||||
description: "Extract every raw session into a search-friendly daily note (one note per session)."
|
||||
parameters:
|
||||
type: object
|
||||
properties: { }
|
||||
steps:
|
||||
- backend: clear_paths_step # wipe old daily notes so this is a clean rebuild
|
||||
config_keys: [daily_dir]
|
||||
- backend: lme_auto_memory_step
|
||||
agent_wrapper: lme_memory
|
||||
|
||||
update_index:
|
||||
backend: base
|
||||
watch_dirs: [daily_dir]
|
||||
watch_suffixes: [md]
|
||||
steps:
|
||||
- backend: clear_store_step
|
||||
- backend: init_changes_step
|
||||
monitor_type: file_store
|
||||
monitor_name: default
|
||||
dispatch_steps: [update_index_step]
|
||||
|
||||
extract_session_by_id:
|
||||
backend: base
|
||||
description: "Given a session_id shown in a search result, go back to the original raw session and extract everything in it that is relevant to the current question. Use this when a search hit looks relevant but the distilled note lacks an exact number, date, or wording."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
session_id:
|
||||
type: string
|
||||
description: "the session_id from a search result header"
|
||||
required:
|
||||
- session_id
|
||||
steps:
|
||||
- backend: lme_extract_session_step
|
||||
agent_wrapper: lme_extract
|
||||
|
||||
vector_search:
|
||||
backend: base
|
||||
description: "Dense semantic search over the memory notes. Returns each hit with its source note path and session_id."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
query:
|
||||
type: string
|
||||
description: "search query"
|
||||
required:
|
||||
- query
|
||||
steps:
|
||||
- backend: vector_search_step
|
||||
include_source: true
|
||||
|
||||
bm25_search:
|
||||
backend: base
|
||||
description: "Keyword (BM25) search over the memory notes. Returns each hit with its source note path and session_id."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
query:
|
||||
type: string
|
||||
description: "search query"
|
||||
required:
|
||||
- query
|
||||
steps:
|
||||
- backend: bm25_search_step
|
||||
include_source: true
|
||||
|
||||
agentic_answer:
|
||||
backend: base
|
||||
description: "Answer query.json from indexed memory using vector/bm25 search + session extraction."
|
||||
parameters:
|
||||
type: object
|
||||
properties: { }
|
||||
steps:
|
||||
- backend: clear_paths_step # drop the previous answer so each run rewrites cleanly
|
||||
paths: [mem_answer.json]
|
||||
- backend: lme_agentic_answer_step
|
||||
agent_wrapper: lme_agentic_answer
|
||||
|
||||
llm_judge:
|
||||
backend: base
|
||||
description: "Judge mem_answer.json against answer.json and write the judgement back into mem_answer.json."
|
||||
parameters:
|
||||
type: object
|
||||
properties: { }
|
||||
steps:
|
||||
- backend: lme_llm_judge_step
|
||||
agent_wrapper: lme_judge
|
||||
|
||||
version:
|
||||
backend: base
|
||||
description: "return reme package version"
|
||||
parameters:
|
||||
type: object
|
||||
properties: { }
|
||||
steps:
|
||||
- backend: version_step
|
||||
|
||||
search:
|
||||
backend: base
|
||||
description: "Hybrid workspace search (vector + BM25, RRF-fused)."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
query:
|
||||
type: string
|
||||
description: "search query"
|
||||
start_date:
|
||||
type: string
|
||||
description: "optional inclusive start date filter (YYYY-MM-DD); results earlier than this date are excluded"
|
||||
end_date:
|
||||
type: string
|
||||
description: "optional inclusive end date filter (YYYY-MM-DD); results later than this date are excluded"
|
||||
required:
|
||||
- query
|
||||
steps:
|
||||
- backend: search_step
|
||||
vector_weight: 0.7
|
||||
candidate_multiplier: 5.0
|
||||
expand_links: true
|
||||
max_links_per_direction: 10
|
||||
|
||||
add_draft:
|
||||
backend: base
|
||||
description: "Append text to the current draft list."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
text:
|
||||
type: string
|
||||
description: "draft text to append"
|
||||
required:
|
||||
- text
|
||||
steps:
|
||||
- backend: add_draft_step
|
||||
|
||||
read_all_draft:
|
||||
backend: base
|
||||
description: "Read all draft text previously appended in the current tool context."
|
||||
parameters:
|
||||
type: object
|
||||
properties: { }
|
||||
steps:
|
||||
- backend: read_all_draft_step
|
||||
|
||||
python_execute:
|
||||
backend: base
|
||||
description: "Execute Python code and return printed stdout."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
code:
|
||||
type: string
|
||||
description: "Python code to execute. Print the final result to stdout."
|
||||
timeout:
|
||||
type: number
|
||||
description: "Execution timeout in seconds; defaults to 60."
|
||||
required:
|
||||
- code
|
||||
steps:
|
||||
- backend: python_execute_step
|
||||
|
||||
session_review:
|
||||
backend: base
|
||||
description: "Review every session for query/answer-relevant evidence and write session_review.json."
|
||||
parameters:
|
||||
type: object
|
||||
properties: { }
|
||||
steps:
|
||||
- backend: clear_paths_step # drop the previous review so each run rewrites cleanly
|
||||
paths: [session_review.json]
|
||||
- backend: lme_session_review_step
|
||||
agent_wrapper: lme_review
|
||||
|
||||
golden_check:
|
||||
backend: base
|
||||
description: "Read session_review.json and judge whether the golden answer is reasonable."
|
||||
parameters:
|
||||
type: object
|
||||
properties: { }
|
||||
steps:
|
||||
- backend: clear_paths_step # drop the previous verdict so each run rewrites cleanly
|
||||
paths: [check_golden.json]
|
||||
- backend: wait_for_paths_step # wait until session_review finishes; comment this step to fail fast instead
|
||||
paths: [session_review.json]
|
||||
poll_seconds: 5
|
||||
log_every_seconds: 60
|
||||
- backend: lme_golden_check_step
|
||||
agent_wrapper: lme_judge
|
||||
|
||||
final_answer_review:
|
||||
backend: base
|
||||
description: "Review one LongMemEval golden answer from all sessions available by question_date."
|
||||
parameters:
|
||||
type: object
|
||||
properties: { }
|
||||
steps:
|
||||
- backend: lme_final_answer_review_step
|
||||
agent_wrapper: lme_final_answer_review
|
||||
reference_paths:
|
||||
- benchmark/longmemeval/golden_check_list_false.jsonl
|
||||
- benchmark/longmemeval/merge_confirm_jinli_false.jsonl
|
||||
retry_initial_seconds: 5
|
||||
retry_max_seconds: 300
|
||||
|
||||
components:
|
||||
tokenizer:
|
||||
default:
|
||||
backend: regex
|
||||
|
||||
as_embedding:
|
||||
default:
|
||||
backend: ${EMBEDDING_BACKEND:-openai}
|
||||
model: ${EMBEDDING_MODEL_NAME:-text-embedding-v4}
|
||||
dimensions: 1024
|
||||
credential:
|
||||
api_key: ${EMBEDDING_API_KEY:-}
|
||||
base_url: ${EMBEDDING_BASE_URL:-https://dashscope.aliyuncs.com/compatible-mode/v1}
|
||||
parameters: { }
|
||||
|
||||
embedding_store:
|
||||
default:
|
||||
backend: local
|
||||
as_embedding: default
|
||||
|
||||
as_llm:
|
||||
default:
|
||||
backend: ${LLM_BACKEND:-openai}
|
||||
model: ${LLM_MODEL_NAME:-qwen3.7-max}
|
||||
stream: true
|
||||
context_size: 1000000
|
||||
max_retries: 3
|
||||
credential:
|
||||
api_key: ${LLM_API_KEY:-}
|
||||
base_url: ${LLM_BASE_URL:-}
|
||||
parameters:
|
||||
max_tokens: 65536
|
||||
|
||||
plus:
|
||||
backend: ${LLM_BACKEND:-openai}
|
||||
model: qwen3.7-plus
|
||||
stream: true
|
||||
context_size: 1000000
|
||||
max_retries: 3
|
||||
credential:
|
||||
api_key: ${LLM_API_KEY:-}
|
||||
base_url: ${LLM_BASE_URL:-}
|
||||
parameters:
|
||||
max_tokens: 65536
|
||||
|
||||
agent_wrapper:
|
||||
default:
|
||||
backend: agentscope
|
||||
as_llm: default
|
||||
permission_mode: bypass
|
||||
react_config:
|
||||
max_iters: 30
|
||||
context_config:
|
||||
trigger_ratio: 0.89
|
||||
reserve_ratio: 0.1
|
||||
tool_result_limit: 50000
|
||||
model_config:
|
||||
max_retries: 3
|
||||
|
||||
agentic_search_agentwrapper:
|
||||
backend: agentscope
|
||||
as_llm: default
|
||||
cwd: session
|
||||
permission_mode: bypass
|
||||
builtin_tools: false
|
||||
job_tools:
|
||||
- search
|
||||
- add_draft
|
||||
- read_all_draft
|
||||
- python_execute
|
||||
sequential_tool_calls: true
|
||||
react_config:
|
||||
max_iters: 100
|
||||
context_config:
|
||||
trigger_ratio: 0.89
|
||||
reserve_ratio: 0.1
|
||||
tool_result_limit: 1000000
|
||||
model_config:
|
||||
max_retries: 3
|
||||
|
||||
lme_memory:
|
||||
backend: agentscope
|
||||
as_llm: plus
|
||||
permission_mode: bypass
|
||||
builtin_tools: false
|
||||
react_config:
|
||||
max_iters: 8
|
||||
context_config:
|
||||
trigger_ratio: 0.89
|
||||
reserve_ratio: 0.1
|
||||
tool_result_limit: 1000000
|
||||
model_config:
|
||||
max_retries: 3
|
||||
|
||||
lme_extract:
|
||||
backend: agentscope
|
||||
as_llm: plus
|
||||
permission_mode: bypass
|
||||
builtin_tools: false
|
||||
react_config:
|
||||
max_iters: 3
|
||||
context_config:
|
||||
trigger_ratio: 0.89
|
||||
reserve_ratio: 0.1
|
||||
tool_result_limit: 1000000
|
||||
model_config:
|
||||
max_retries: 3
|
||||
|
||||
lme_agentic_answer:
|
||||
backend: agentscope
|
||||
as_llm: default
|
||||
cwd: session
|
||||
permission_mode: bypass
|
||||
builtin_tools: false
|
||||
job_tools:
|
||||
- vector_search
|
||||
- bm25_search
|
||||
- python_execute
|
||||
- extract_session_by_id
|
||||
sequential_tool_calls: true
|
||||
react_config:
|
||||
max_iters: 40
|
||||
context_config:
|
||||
trigger_ratio: 0.89
|
||||
reserve_ratio: 0.1
|
||||
tool_result_limit: 1000000
|
||||
model_config:
|
||||
max_retries: 3
|
||||
|
||||
claude_code:
|
||||
backend: claude_code
|
||||
model: ${CLAUDE_CODE_MODEL_NAME:-glm-5.2}
|
||||
api_key: ${CLAUDE_CODE_API_KEY:-}
|
||||
base_url: ${CLAUDE_CODE_BASE_URL:-https://dashscope.aliyuncs.com/apps/anthropic}
|
||||
permission_mode: bypassPermissions
|
||||
|
||||
lme_final_answer_review:
|
||||
backend: claude_code
|
||||
model: ${CLAUDE_CODE_MODEL_NAME:-claude-opus-4-8}
|
||||
api_key: ${CLAUDE_CODE_API_KEY:-}
|
||||
base_url: ${CLAUDE_CODE_BASE_URL:-}
|
||||
cwd: session
|
||||
permission_mode: bypassPermissions
|
||||
|
||||
lme_review:
|
||||
backend: agentscope
|
||||
as_llm: plus
|
||||
permission_mode: bypass
|
||||
builtin_tools: false
|
||||
react_config:
|
||||
max_iters: 10
|
||||
context_config:
|
||||
trigger_ratio: 0.89
|
||||
reserve_ratio: 0.1
|
||||
tool_result_limit: 1000000
|
||||
model_config:
|
||||
max_retries: 3
|
||||
|
||||
lme_judge:
|
||||
backend: agentscope
|
||||
as_llm: default
|
||||
permission_mode: bypass
|
||||
builtin_tools: false
|
||||
job_tools:
|
||||
- python_execute
|
||||
sequential_tool_calls: true
|
||||
react_config:
|
||||
max_iters: 50
|
||||
context_config:
|
||||
trigger_ratio: 0.89
|
||||
reserve_ratio: 0.1
|
||||
tool_result_limit: 1000000
|
||||
model_config:
|
||||
max_retries: 3
|
||||
|
||||
file_graph:
|
||||
default:
|
||||
backend: local
|
||||
|
||||
file_chunker:
|
||||
markdown:
|
||||
backend: markdown
|
||||
supported_extensions: [ "md" ]
|
||||
embed_toc: true
|
||||
max_ast_sections: 100
|
||||
default:
|
||||
backend: default
|
||||
supported_extensions: [ "json", "jsonl" ]
|
||||
chunk_byte_size: 100000
|
||||
|
||||
keyword_index:
|
||||
default:
|
||||
backend: bm25
|
||||
tokenizer: default
|
||||
|
||||
file_store:
|
||||
default:
|
||||
backend: local
|
||||
store_name: local
|
||||
embedding_store: default
|
||||
# embedding_store: ""
|
||||
keyword_index: default
|
||||
file_graph: default
|
||||
583
reme/config/lme.yaml
Normal file
583
reme/config/lme.yaml
Normal file
|
|
@ -0,0 +1,583 @@
|
|||
# LongMemEval benchmark config — based on longmemeval.yaml (split)
|
||||
# All background/cron jobs are converted to base (manually callable).
|
||||
# Use with: resolve_app_config(config="lme.yaml", ...)
|
||||
|
||||
service:
|
||||
backend: http
|
||||
|
||||
jobs:
|
||||
# ── LongMemEval agentic answer (ReAct agent + search tool) ──
|
||||
agentic_answer:
|
||||
backend: base
|
||||
description: "LongMemEval agentic answer job (ReAct agent with search tool)"
|
||||
watch_dirs: []
|
||||
watch_suffixes: []
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
query:
|
||||
type: string
|
||||
description: "The query to ask"
|
||||
query_time:
|
||||
type: string
|
||||
description: "ISO timestamp representing the query time"
|
||||
default: ""
|
||||
required:
|
||||
- query
|
||||
steps:
|
||||
- backend: lme_agentic_answer_step
|
||||
agent_wrapper: bench
|
||||
|
||||
# ── LLM-as-Judge for evaluating answers ──
|
||||
answer_judge:
|
||||
backend: base
|
||||
description: "LLM-as-Judge: evaluate agent answer against golden answer"
|
||||
watch_dirs: []
|
||||
watch_suffixes: []
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
query:
|
||||
type: string
|
||||
description: "The question being asked"
|
||||
agent_answer:
|
||||
type: string
|
||||
description: "The model's answer to evaluate"
|
||||
golden_answer:
|
||||
type: string
|
||||
description: "The correct/golden answer"
|
||||
question_type:
|
||||
type: string
|
||||
description: "The question type for prompt selection"
|
||||
default: ""
|
||||
required:
|
||||
- query
|
||||
- agent_answer
|
||||
- golden_answer
|
||||
steps:
|
||||
- backend: lme_answer_judge_step
|
||||
agent_wrapper: judge
|
||||
|
||||
# ── Manual index update (replaces index_update_loop background) ──
|
||||
index_update:
|
||||
backend: base
|
||||
description: "Manually trigger incremental index update for watched dirs."
|
||||
watch_dirs: [daily_dir, digest_dir, dialog_dir]
|
||||
watch_suffixes: [md, jsonl]
|
||||
parameters:
|
||||
type: object
|
||||
properties: {}
|
||||
steps:
|
||||
- backend: init_changes_step
|
||||
monitor_type: file_store
|
||||
monitor_name: default
|
||||
dispatch_steps: [update_index_step]
|
||||
|
||||
# ── Manual digest catalog update (replaces digest_watch_loop background) ──
|
||||
digest_update:
|
||||
backend: base
|
||||
description: "Manually trigger digest catalog update."
|
||||
watch_dirs: [daily_dir, digest_dir]
|
||||
watch_suffixes: [md]
|
||||
parameters:
|
||||
type: object
|
||||
properties: {}
|
||||
steps:
|
||||
- backend: init_changes_step
|
||||
monitor_type: file_catalog
|
||||
monitor_name: digest
|
||||
dispatch_steps:
|
||||
- backend: update_catalog_step
|
||||
file_catalog: digest
|
||||
- backend: log_changes_step
|
||||
|
||||
# ── Auto dream (same as default.yaml auto_dream, base mode) ──
|
||||
# auto_dream:
|
||||
# backend: base
|
||||
# description: "Auto-dream: scan today's day-index and daily notes, globally extract merged units/topics, integrate digest units, write interests.yaml, and persist the dream catalog."
|
||||
# parameters:
|
||||
# type: object
|
||||
# properties:
|
||||
# date:
|
||||
# type: string
|
||||
# description: "YYYY-MM-DD to scan; defaults to today in the dreamer's timezone"
|
||||
# default: ""
|
||||
# hint:
|
||||
# type: string
|
||||
# description: "caller guidance passed through to dream extract/integrate"
|
||||
# default: ""
|
||||
# scan_days:
|
||||
# type: integer
|
||||
# description: "number of recent daily directories to scan, ending at date"
|
||||
# default: 2
|
||||
# max_units:
|
||||
# type: integer
|
||||
# description: "maximum number of extracted memory units"
|
||||
# default: 5
|
||||
# topic_count:
|
||||
# type: integer
|
||||
# description: "maximum number of final daily interest topics"
|
||||
# default: 3
|
||||
# topic_diversity_days:
|
||||
# type: integer
|
||||
# description: "number of previous interests.yaml days to avoid repeating"
|
||||
# default: 7
|
||||
# steps:
|
||||
# - backend: dream_extract_step
|
||||
# file_catalog: dream
|
||||
# topic_session_id: interests
|
||||
# scan_days: 2
|
||||
# max_units: 5
|
||||
# - backend: dream_integrate_step
|
||||
# - backend: dream_topics_step
|
||||
# topic_count: 3
|
||||
# topic_diversity_days: 7
|
||||
# - backend: dream_finish_step
|
||||
# file_catalog: dream
|
||||
|
||||
# ── Auto memory (same as default.yaml) ──
|
||||
auto_memory:
|
||||
backend: base
|
||||
description: "Auto-memory: record conversation facts into a daily note"
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
messages:
|
||||
type: array
|
||||
description: "messages"
|
||||
items:
|
||||
type: object
|
||||
session_id:
|
||||
type: string
|
||||
description: "source conversation session identifier"
|
||||
default: ""
|
||||
memory_hint:
|
||||
type: string
|
||||
description: "optional hint"
|
||||
date:
|
||||
type: string
|
||||
description: "YYYY-MM-DD daily note date; empty = infer from message timestamps or today"
|
||||
default: ""
|
||||
required:
|
||||
- messages
|
||||
steps:
|
||||
- backend: lme_auto_memory_step
|
||||
|
||||
# ── Reindex (full rebuild) ──
|
||||
reindex:
|
||||
backend: base
|
||||
description: "wipe the file store and rebuild it from the existing files"
|
||||
watch_dirs: [daily_dir, digest_dir, resource_dir]
|
||||
watch_suffixes: [md, jsonl]
|
||||
parameters:
|
||||
type: object
|
||||
properties: {}
|
||||
steps:
|
||||
- backend: clear_store_step
|
||||
- backend: init_changes_step
|
||||
monitor_type: file_store
|
||||
monitor_name: default
|
||||
dispatch_steps: [update_index_step]
|
||||
|
||||
# ── Search ──
|
||||
# start_date:
|
||||
# type: string
|
||||
# description: "optional inclusive start date filter (YYYY-MM-DD); results earlier than this date are excluded"
|
||||
# end_date:
|
||||
# type: string
|
||||
# description: "optional inclusive end date filter (YYYY-MM-DD); results later than this date are excluded"
|
||||
|
||||
search:
|
||||
backend: base
|
||||
description: "Hybrid workspace search (vector + BM25, RRF-fused) with deduplication."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
query:
|
||||
type: string
|
||||
description: "search query"
|
||||
start_date:
|
||||
type: string
|
||||
description: "optional inclusive start date filter (YYYY-MM-DD); results earlier than this date are excluded"
|
||||
end_date:
|
||||
type: string
|
||||
description: "optional inclusive end date filter (YYYY-MM-DD); results later than this date are excluded"
|
||||
# vector_weight:
|
||||
# type: number
|
||||
# description: >-
|
||||
# Optional weight balancing vector similarity vs BM25 keyword matching in the
|
||||
# RRF fusion. Recommended value is 0.7, which provides a good balance between
|
||||
# semantic (vector) similarity and lexical (BM25) matching. Values close to 0
|
||||
# emphasize BM25 keyword matching, values close to 1 emphasize vector semantic
|
||||
# similarity.
|
||||
required:
|
||||
- query
|
||||
steps:
|
||||
- backend: search_step
|
||||
vector_weight: 0.7
|
||||
candidate_multiplier: 5.0
|
||||
expand_links: true
|
||||
max_links_per_direction: 10
|
||||
|
||||
add_draft:
|
||||
backend: base
|
||||
description: "Append text to the current draft list."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
text:
|
||||
type: string
|
||||
description: "draft text to append"
|
||||
required:
|
||||
- text
|
||||
steps:
|
||||
- backend: add_draft_step
|
||||
|
||||
read_all_draft:
|
||||
backend: base
|
||||
description: "Read all draft text previously appended in the current tool context."
|
||||
parameters:
|
||||
type: object
|
||||
properties: { }
|
||||
steps:
|
||||
- backend: read_all_draft_step
|
||||
|
||||
python_execute:
|
||||
backend: base
|
||||
description: "Execute Python code and return printed stdout."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
code:
|
||||
type: string
|
||||
description: "Python code to execute. Print the final result to stdout."
|
||||
timeout:
|
||||
type: number
|
||||
description: "Execution timeout in seconds; defaults to 60."
|
||||
required:
|
||||
- code
|
||||
steps:
|
||||
- backend: python_execute_step
|
||||
|
||||
# ── File I/O jobs (needed by auto_memory agent tools) ──
|
||||
daily_list:
|
||||
backend: base
|
||||
description: "List notes under a single day."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
date:
|
||||
type: string
|
||||
description: "YYYY-MM-DD; empty = today"
|
||||
default: ""
|
||||
steps:
|
||||
- backend: daily_list_step
|
||||
|
||||
daily_reindex:
|
||||
backend: base
|
||||
description: "Rebuild the day-index page daily/<date>.md."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
date:
|
||||
type: string
|
||||
description: "YYYY-MM-DD; empty = today"
|
||||
default: ""
|
||||
steps:
|
||||
- backend: daily_reindex_step
|
||||
|
||||
frontmatter_update:
|
||||
backend: base
|
||||
description: "Merge key-values into a file's frontmatter."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
path:
|
||||
type: string
|
||||
description: "workspace-relative path"
|
||||
metadata:
|
||||
type: object
|
||||
description: "key-values to merge"
|
||||
required:
|
||||
- path
|
||||
- metadata
|
||||
steps:
|
||||
- backend: frontmatter_update_step
|
||||
|
||||
move:
|
||||
backend: base
|
||||
description: "Move / rename a workspace file."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
src_path:
|
||||
type: string
|
||||
description: "workspace-relative source"
|
||||
dst_path:
|
||||
type: string
|
||||
description: "workspace-relative destination"
|
||||
overwrite:
|
||||
type: boolean
|
||||
default: false
|
||||
retarget:
|
||||
type: boolean
|
||||
default: true
|
||||
required:
|
||||
- src_path
|
||||
- dst_path
|
||||
steps:
|
||||
- backend: move_step
|
||||
|
||||
read:
|
||||
backend: base
|
||||
description: "Read a markdown file under the workspace."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
path:
|
||||
type: string
|
||||
description: "workspace-relative path; markdown only"
|
||||
start_line:
|
||||
type: integer
|
||||
end_line:
|
||||
type: integer
|
||||
required:
|
||||
- path
|
||||
steps:
|
||||
- backend: read_step
|
||||
with_neighbors: false
|
||||
max_neighbors_per_direction: 10
|
||||
|
||||
write:
|
||||
backend: base
|
||||
description: "Write a markdown file."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
path:
|
||||
type: string
|
||||
name:
|
||||
type: string
|
||||
description:
|
||||
type: string
|
||||
content:
|
||||
type: string
|
||||
metadata:
|
||||
type: object
|
||||
required:
|
||||
- path
|
||||
- name
|
||||
- description
|
||||
- content
|
||||
steps:
|
||||
- backend: write_step
|
||||
|
||||
daily_write:
|
||||
backend: base
|
||||
description: "Write a daily markdown note."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
name:
|
||||
type: string
|
||||
description:
|
||||
type: string
|
||||
session_id:
|
||||
type: string
|
||||
content:
|
||||
type: string
|
||||
date:
|
||||
type: string
|
||||
default: ""
|
||||
metadata:
|
||||
type: object
|
||||
required:
|
||||
- name
|
||||
- description
|
||||
- session_id
|
||||
- content
|
||||
steps:
|
||||
- backend: daily_write_step
|
||||
|
||||
edit:
|
||||
backend: base
|
||||
description: "Find-and-replace in a markdown file."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
path:
|
||||
type: string
|
||||
old:
|
||||
type: string
|
||||
new:
|
||||
type: string
|
||||
default: ""
|
||||
required:
|
||||
- path
|
||||
- old
|
||||
- new
|
||||
steps:
|
||||
- backend: edit_step
|
||||
|
||||
frontmatter_read:
|
||||
backend: base
|
||||
description: "Read a file's frontmatter as a dict."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
path:
|
||||
type: string
|
||||
required:
|
||||
- path
|
||||
steps:
|
||||
- backend: frontmatter_read_step
|
||||
|
||||
node_search:
|
||||
backend: base
|
||||
description: "Digest node recall."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
query:
|
||||
type: string
|
||||
limit:
|
||||
type: integer
|
||||
default: 20
|
||||
required:
|
||||
- query
|
||||
steps:
|
||||
- backend: node_search_step
|
||||
vector_weight: 0.7
|
||||
candidate_multiplier: 5.0
|
||||
|
||||
components:
|
||||
tokenizer:
|
||||
default:
|
||||
backend: regex
|
||||
|
||||
as_embedding:
|
||||
default:
|
||||
backend: ${EMBEDDING_BACKEND:-openai}
|
||||
model: ${EMBEDDING_MODEL_NAME:-text-embedding-v4}
|
||||
credential:
|
||||
api_key: ${EMBEDDING_API_KEY:-}
|
||||
base_url: ${EMBEDDING_BASE_URL:-https://dashscope.aliyuncs.com/compatible-mode/v1}
|
||||
dimensions: 1024
|
||||
|
||||
embedding_store:
|
||||
default:
|
||||
backend: local
|
||||
as_embedding: default
|
||||
|
||||
as_llm:
|
||||
default:
|
||||
backend: ${LLM_BACKEND:-openai}
|
||||
model: ${LLM_MODEL_NAME:-qwen3.6-flash}
|
||||
stream: true
|
||||
context_size: 200000
|
||||
max_retries: 3
|
||||
retry_delay: 5.0
|
||||
credential:
|
||||
api_key: ${LLM_API_KEY:-}
|
||||
base_url: ${LLM_BASE_URL:-}
|
||||
parameters:
|
||||
max_tokens: 65536
|
||||
thinking_enable: false
|
||||
judge:
|
||||
backend: ${LLM_BACKEND:-openai}
|
||||
model: ${JUDGE_MODEL_NAME:-qwen3.7-max}
|
||||
stream: false
|
||||
context_size: 200000
|
||||
max_retries: 5
|
||||
credential:
|
||||
api_key: ${LLM_API_KEY:-}
|
||||
base_url: ${LLM_BASE_URL:-}
|
||||
parameters:
|
||||
max_tokens: 65536
|
||||
thinking_enable: false
|
||||
bench:
|
||||
backend: ${LLM_BACKEND:-openai}
|
||||
model: ${BENCH_MODEL_NAME:-qwen3.7-max}
|
||||
stream: true
|
||||
context_size: 400000
|
||||
max_retries: 5
|
||||
retry_delay: 5.0
|
||||
credential:
|
||||
api_key: ${LLM_API_KEY:-}
|
||||
base_url: ${LLM_BASE_URL:-}
|
||||
parameters:
|
||||
max_tokens: 65536
|
||||
thinking_enable: true
|
||||
|
||||
agent_wrapper:
|
||||
default:
|
||||
backend: agentscope
|
||||
as_llm: default
|
||||
permission_mode: bypass
|
||||
react_config:
|
||||
max_iters: 30
|
||||
context_config:
|
||||
trigger_ratio: 0.8
|
||||
reserve_ratio: 0.1
|
||||
tool_result_limit: 50000
|
||||
model_config:
|
||||
max_retries: 1
|
||||
judge:
|
||||
backend: agentscope
|
||||
as_llm: judge
|
||||
permission_mode: bypass
|
||||
react_config:
|
||||
max_iters: 1
|
||||
context_config:
|
||||
trigger_ratio: 0.8
|
||||
reserve_ratio: 0.1
|
||||
tool_result_limit: 50000
|
||||
model_config:
|
||||
max_retries: 1
|
||||
bench:
|
||||
backend: agentscope
|
||||
as_llm: bench
|
||||
permission_mode: bypass
|
||||
react_config:
|
||||
max_iters: 30
|
||||
context_config:
|
||||
trigger_ratio: 0.8
|
||||
reserve_ratio: 0.1
|
||||
tool_result_limit: 50000
|
||||
model_config:
|
||||
max_retries: 1
|
||||
|
||||
file_graph:
|
||||
default:
|
||||
backend: local
|
||||
|
||||
file_catalog:
|
||||
default:
|
||||
backend: local
|
||||
resource:
|
||||
backend: local
|
||||
digest:
|
||||
backend: local
|
||||
dream:
|
||||
backend: local
|
||||
|
||||
file_chunker:
|
||||
markdown:
|
||||
backend: markdown
|
||||
supported_extensions: ["md"]
|
||||
default:
|
||||
backend: default
|
||||
supported_extensions: ["jsonl"]
|
||||
|
||||
keyword_index:
|
||||
default:
|
||||
backend: bm25
|
||||
tokenizer: default
|
||||
|
||||
file_store:
|
||||
default:
|
||||
backend: local
|
||||
store_name: local
|
||||
embedding_store: default
|
||||
keyword_index: default
|
||||
file_graph: default
|
||||
|
|
@ -39,6 +39,7 @@ class ApplicationConfig(BaseModel):
|
|||
resource_dir: str = Field(default="resource", description="Subdirectory for external assets")
|
||||
daily_dir: str = Field(default="daily", description="Subdirectory for daily memory")
|
||||
digest_dir: str = Field(default="digest", description="Subdirectory for digest memory")
|
||||
dialog_dir: str = Field(default="session/dialog", description="Subdirectory for dialog session transcripts")
|
||||
enable_logo: bool = Field(default=True, description="Show ASCII logo on startup")
|
||||
timezone: str | None = Field(default="Asia/Shanghai", description="IANA timezone; None uses local time")
|
||||
language: str = Field(default="", description="Default language for LLM interactions")
|
||||
|
|
|
|||
|
|
@ -1,12 +1,17 @@
|
|||
"""Benchmark steps."""
|
||||
|
||||
from . import lme
|
||||
from .lme import ContextAnswerStep, GoldenCheckStep, LmeLlmJudgeStep, SessionReviewStep
|
||||
from . import base, lme, beam
|
||||
from .base import BaseAgenticAnswerStep
|
||||
from .lme import LmeAgenticAnswerStep, LmeAnswerJudgeStep
|
||||
from .beam import BeamAgenticAnswerStep, BeamRubricJudgeStep
|
||||
|
||||
__all__ = [
|
||||
"ContextAnswerStep",
|
||||
"GoldenCheckStep",
|
||||
"LmeLlmJudgeStep",
|
||||
"SessionReviewStep",
|
||||
"BaseAgenticAnswerStep",
|
||||
"LmeAgenticAnswerStep",
|
||||
"LmeAnswerJudgeStep",
|
||||
"BeamAgenticAnswerStep",
|
||||
"BeamRubricJudgeStep",
|
||||
"base",
|
||||
"lme",
|
||||
"beam",
|
||||
]
|
||||
|
|
|
|||
7
reme/steps/benchmark/base/__init__.py
Normal file
7
reme/steps/benchmark/base/__init__.py
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
"""Shared base classes for benchmark steps."""
|
||||
|
||||
from .agentic_answer import BaseAgenticAnswerStep
|
||||
|
||||
__all__ = [
|
||||
"BaseAgenticAnswerStep",
|
||||
]
|
||||
102
reme/steps/benchmark/base/agentic_answer.py
Normal file
102
reme/steps/benchmark/base/agentic_answer.py
Normal file
|
|
@ -0,0 +1,102 @@
|
|||
"""Shared base class for benchmark agentic-answer steps."""
|
||||
|
||||
import os
|
||||
import threading
|
||||
|
||||
from ...base_step import BaseStep
|
||||
from ....enumeration import ChunkEnum
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Process-safe & thread-safe counter for unique tool_context_id.
|
||||
# PID guarantees cross-process uniqueness (multiprocessing Pool);
|
||||
# threading.Lock + monotonic counter guarantees thread safety within a process.
|
||||
# ---------------------------------------------------------------------------
|
||||
_TOOL_CTX_LOCK = threading.Lock()
|
||||
_TOOL_CTX_SEQ = 0
|
||||
|
||||
|
||||
def _next_tool_context_id(prefix: str) -> str:
|
||||
global _TOOL_CTX_SEQ
|
||||
with _TOOL_CTX_LOCK:
|
||||
_TOOL_CTX_SEQ += 1
|
||||
seq = _TOOL_CTX_SEQ
|
||||
return f"{prefix}_{os.getpid()}_{seq}"
|
||||
|
||||
|
||||
class BaseAgenticAnswerStep(BaseStep):
|
||||
"""Base ReAct-agent answer step shared by BEAM and LongMemEval benchmarks.
|
||||
|
||||
Subclasses only need to set:
|
||||
TOOL_CONTEXT_PREFIX (str): prefix used to build the unique tool_context_id.
|
||||
|
||||
And apply their own ``@R.register(...)`` decorator and docstring.
|
||||
|
||||
Inputs (from RuntimeContext):
|
||||
query (str, required): The question to answer.
|
||||
query_time (str, optional): ISO timestamp representing the query time,
|
||||
used to ground the agent's temporal context.
|
||||
|
||||
Output (written to context.response.answer):
|
||||
The agent's final answer text.
|
||||
"""
|
||||
|
||||
MAX_ITERATION = 10
|
||||
TOOL_CONTEXT_PREFIX: str = "agentic_answer"
|
||||
|
||||
async def execute(self):
|
||||
assert self.context is not None
|
||||
query: str = self.context.get("query", "")
|
||||
query_time: str | None = self.context.get("query_time")
|
||||
|
||||
if not query:
|
||||
self.context.response.success = False
|
||||
self.context.response.answer = "Skipped: empty query"
|
||||
return self.context.response
|
||||
|
||||
# Build system prompt with optional temporal context
|
||||
sys_prompt = self.get_prompt("system_prompt")
|
||||
if query_time:
|
||||
sys_prompt += "\n" + self.prompt_format("temporal_hint", query_time=query_time)
|
||||
|
||||
wrapper_kwargs = {
|
||||
"system_prompt": sys_prompt,
|
||||
"job_tools": ["search", "add_draft", "read_all_draft"],
|
||||
"react_config": {"max_iters": self.MAX_ITERATION},
|
||||
"tool_context_id": _next_tool_context_id(self.TOOL_CONTEXT_PREFIX),
|
||||
}
|
||||
|
||||
if self.context.stream:
|
||||
text = await self._stream_reply(query, **wrapper_kwargs)
|
||||
else:
|
||||
result = await self.agent_wrapper.reply(query, **wrapper_kwargs)
|
||||
text = (result.get("result") or "").strip()
|
||||
|
||||
self.logger.debug(f"[{self.name}] response: {text!r}")
|
||||
|
||||
self.context.response.success = True
|
||||
self.context.response.answer = text
|
||||
self.context.response.metadata.update(
|
||||
{
|
||||
"query": query,
|
||||
"query_time": query_time,
|
||||
"sys_prompt": sys_prompt,
|
||||
"response": text,
|
||||
},
|
||||
)
|
||||
return self.context.response
|
||||
|
||||
async def _stream_reply(self, query: str, **wrapper_kwargs) -> str:
|
||||
"""Stream unified chunks to the context stream queue."""
|
||||
assert self.context is not None
|
||||
text_parts: list[str] = []
|
||||
|
||||
async for chunk in self.agent_wrapper.reply_stream(query, **wrapper_kwargs):
|
||||
await self.context.add_stream_string(chunk.chunk, chunk.chunk_type)
|
||||
|
||||
if chunk.chunk_type == ChunkEnum.CONTENT and isinstance(chunk.chunk, str):
|
||||
text_parts.append(chunk.chunk)
|
||||
|
||||
if chunk.session_id:
|
||||
self.context.response.metadata["session_id"] = chunk.session_id
|
||||
|
||||
return "".join(text_parts).strip()
|
||||
11
reme/steps/benchmark/beam/__init__.py
Normal file
11
reme/steps/benchmark/beam/__init__.py
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
"""BEAM benchmark steps."""
|
||||
|
||||
from .agentic_answer import BeamAgenticAnswerStep
|
||||
from .llm_judge import BeamRubricJudgeStep
|
||||
from .auto_memory import BeamAutoMemoryStep
|
||||
|
||||
__all__ = [
|
||||
"BeamAgenticAnswerStep",
|
||||
"BeamRubricJudgeStep",
|
||||
"BeamAutoMemoryStep",
|
||||
]
|
||||
16
reme/steps/benchmark/beam/agentic_answer.py
Normal file
16
reme/steps/benchmark/beam/agentic_answer.py
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
"""BEAM agentic answer step – ReAct agent that answers questions using the search tool."""
|
||||
|
||||
from ....components import R
|
||||
from ..base import BaseAgenticAnswerStep
|
||||
|
||||
|
||||
@R.register("beam_agentic_answer_step")
|
||||
class BeamAgenticAnswerStep(BaseAgenticAnswerStep):
|
||||
"""Answer a BEAM probing question via ReAct agent with access to the search tool.
|
||||
|
||||
The agent uses the ``agent_wrapper`` component in ReAct mode, calling the
|
||||
``search`` job tool to retrieve relevant memory chunks before generating
|
||||
a final answer.
|
||||
"""
|
||||
|
||||
TOOL_CONTEXT_PREFIX = "beam_agentic_answer"
|
||||
15
reme/steps/benchmark/beam/agentic_answer.yaml
Normal file
15
reme/steps/benchmark/beam/agentic_answer.yaml
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
system_prompt: |
|
||||
You are a memory retrieval assistant. You MUST use the search tool to find information before answering.
|
||||
- Your total time of tool calls should be at most 9 times
|
||||
|
||||
## Search Strategy
|
||||
- You can call 'search' tool to search multiple times (at least once) with different queries to gather comprehensive information.
|
||||
## Draft Tool
|
||||
- Use 'add_draft' to save key findings during search, and 'read_all_draft' to review all saved notes before answering.
|
||||
## Answer Rules
|
||||
- Answer based ONLY on retrieved context.
|
||||
- Output ONLY the direct factual answer — no reasoning, no search process, no elaboration.
|
||||
- If information is not founded or not sufficient after multiple searches, reply: 'Information not found.'
|
||||
|
||||
temporal_hint: |
|
||||
Current time context: {query_time}
|
||||
132
reme/steps/benchmark/beam/auto_memory.py
Normal file
132
reme/steps/benchmark/beam/auto_memory.py
Normal file
|
|
@ -0,0 +1,132 @@
|
|||
"""auto_memory (beam) — AutoMemoryStep with timestamp interpolation and daily_write date default."""
|
||||
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from agentscope.message import Msg
|
||||
|
||||
from ...evolve.auto_memory import AutoMemoryStep, _normalize_msg_timestamp
|
||||
from ....components import R
|
||||
|
||||
|
||||
def _parse_iso_seconds(value: str) -> datetime | None:
|
||||
"""Parse an ISO-8601 timestamp that is precise to at least seconds.
|
||||
|
||||
Accepts formats like:
|
||||
2026-07-01T14:30:00
|
||||
2026-07-01T14:30:00Z
|
||||
2026-07-01T14:30:00+08:00
|
||||
2026-07-01T14:30:00.123456
|
||||
|
||||
Rejects date-only (``2026-07-01``) or minute-only (``2026-07-01T14:30``).
|
||||
Returns ``None`` when the value does not satisfy the requirements.
|
||||
"""
|
||||
text = str(value).strip()
|
||||
# Minimum valid: YYYY-MM-DDTHH:MM:SS = 19 chars
|
||||
if len(text) < 19:
|
||||
return None
|
||||
# Must contain 'T' separator and at least HH:MM:SS after it
|
||||
if "T" not in text:
|
||||
return None
|
||||
time_part = text.split("T", 1)[1]
|
||||
# time_part must start with HH:MM:SS (8 chars minimum)
|
||||
if len(time_part) < 8 or time_part[2] != ":" or time_part[5] != ":":
|
||||
return None
|
||||
try:
|
||||
return datetime.fromisoformat(text)
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def _interpolate_timestamps(items: list[dict]) -> list[dict]:
|
||||
"""Fill missing ``created_at`` fields via linear interpolation.
|
||||
|
||||
Rules (backward-compatible — returns *items* unchanged when no message
|
||||
carries a ``created_at`` value):
|
||||
|
||||
1. If **no** message has ``created_at`` → return as-is (system time used
|
||||
later by AgentScope's ``Msg`` constructor).
|
||||
2. Messages **before** the first timestamped message → inherit the first
|
||||
timestamp.
|
||||
3. Messages **after** the last timestamped message → inherit the last
|
||||
timestamp.
|
||||
4. Messages **between** two timestamped anchors → linearly interpolated.
|
||||
"""
|
||||
# Pass 1: normalize aliases and collect anchors
|
||||
normalized: list[dict] = []
|
||||
anchors: list[tuple[int, datetime]] = [] # (index, parsed_dt)
|
||||
|
||||
for i, raw_item in enumerate(items):
|
||||
if not isinstance(raw_item, dict):
|
||||
normalized.append(raw_item)
|
||||
continue
|
||||
item = _normalize_msg_timestamp(raw_item)
|
||||
normalized.append(item)
|
||||
ca = item.get("created_at")
|
||||
if ca:
|
||||
dt = _parse_iso_seconds(str(ca))
|
||||
if dt is not None:
|
||||
anchors.append((i, dt))
|
||||
|
||||
# No anchors → fully backward-compatible, let Msg use system time
|
||||
if not anchors:
|
||||
return normalized
|
||||
|
||||
# Pass 2: interpolate
|
||||
result: list[dict] = []
|
||||
for i, item in enumerate(normalized):
|
||||
if not isinstance(item, dict):
|
||||
result.append(item)
|
||||
continue
|
||||
# Already has a valid parsed anchor — keep it
|
||||
if any(idx == i for idx, _ in anchors):
|
||||
result.append(item)
|
||||
continue
|
||||
|
||||
# Find the nearest preceding and following anchors
|
||||
prev_anchor: tuple[int, datetime] | None = None
|
||||
next_anchor: tuple[int, datetime] | None = None
|
||||
for idx, dt in anchors:
|
||||
if idx < i:
|
||||
prev_anchor = (idx, dt)
|
||||
for idx, dt in anchors:
|
||||
if idx > i:
|
||||
next_anchor = (idx, dt)
|
||||
break
|
||||
|
||||
# Determine interpolated time
|
||||
if prev_anchor is None:
|
||||
# Before the first anchor
|
||||
interpolated_dt = anchors[0][1]
|
||||
elif next_anchor is None:
|
||||
# After the last anchor
|
||||
interpolated_dt = anchors[-1][1]
|
||||
else:
|
||||
# Between two anchors — linear
|
||||
prev_idx, prev_dt = prev_anchor
|
||||
next_idx, next_dt = next_anchor
|
||||
span = next_idx - prev_idx
|
||||
ratio = (i - prev_idx) / span
|
||||
delta_seconds = (next_dt - prev_dt).total_seconds()
|
||||
interpolated_dt = prev_dt + timedelta(seconds=delta_seconds * ratio)
|
||||
|
||||
item = {**item, "created_at": interpolated_dt.isoformat()}
|
||||
result.append(item)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
@R.register("beam_auto_memory_step")
|
||||
class BeamAutoMemoryStep(AutoMemoryStep):
|
||||
"""AutoMemoryStep variant that interpolates timestamps and pins daily_write to the resolved day."""
|
||||
|
||||
def _build_messages(self, raw_messages: list) -> list[Msg]:
|
||||
# Interpolate timestamps: if any message carries created_at, fill in
|
||||
# the rest via linear interpolation so the whole session has coherent
|
||||
# time ordering (see _interpolate_timestamps docstring for rules).
|
||||
interpolated = _interpolate_timestamps(
|
||||
[item if not isinstance(item, dict) else dict(item) for item in raw_messages],
|
||||
)
|
||||
return [self._to_msg(item) for item in interpolated]
|
||||
|
||||
def _reply_extra_kwargs(self, day: str) -> dict:
|
||||
return {"tool_defaults": {"daily_write": {"date": day}}}
|
||||
334
reme/steps/benchmark/beam/auto_memory.yaml
Normal file
334
reme/steps/benchmark/beam/auto_memory.yaml
Normal file
|
|
@ -0,0 +1,334 @@
|
|||
system_prompt: |
|
||||
You are an automatic memory extraction system. Your task is to extract ALL valuable information from conversations and record them into a structured daily memory note. Ensure **complete coverage** — every piece of information worth preserving must be captured.
|
||||
|
||||
## Extraction Principles
|
||||
|
||||
1. **Complete Coverage**: Generate enough content to ensure ALL information in the dialogues is captured. Missing information is worse than redundancy.
|
||||
2. **Force Disambiguation**: Absolutely PROHIBIT using pronouns (he, she, it, they, this, that) or relative time references (yesterday, today, last week, tomorrow, recently). Always use full names, explicit dates, and specific references.
|
||||
3. **Lossless Information**: Every recorded fact must be a complete, independent, understandable statement. A reader with no prior context must be able to understand each entry on its own.
|
||||
4. **Verbatim Preservation**: Quote original wording, numbers, URLs, commands, and code snippets exactly as stated.
|
||||
|
||||
## What to Extract
|
||||
|
||||
- **Persons**: All person names mentioned (full names when available)
|
||||
- **Entities**: Companies, products, organizations, projects, repositories
|
||||
- **Decisions & Events**: What happened, what was decided, and why
|
||||
- **Facts & Preferences**: Persistent facts about the user — identity, habits, preferences, constraints
|
||||
- **Procedures & Solutions**: Actionable steps, commands, configurations that can be directly reused
|
||||
- **Current State**: Progress, blockers, next steps, open questions
|
||||
- **Temporal Information**: Use absolute dates in ISO 8601 format (YYYY-MM-DD) when explicit time is mentioned
|
||||
- **Locations**: Specific location names when mentioned
|
||||
- **Topic Keywords**: Core keywords that identify the subject matter
|
||||
|
||||
## Body Format
|
||||
|
||||
Use structured sections with headings. Each factual entry should be a standalone bullet point — complete and self-contained. Group entries by topic. Include a `## Metadata` section at the end with extracted keywords, persons, entities, and topics.
|
||||
|
||||
## Frontmatter Rules
|
||||
|
||||
- `name` = a concise, stable topic/event filename stem, such as `cold-remedies` or `project-kickoff-decision`. Do not include today's date or the daily directory date; the outer daily path already records the date. For existing notes, update it when a better filename is clearly warranted.
|
||||
- `description` = a thorough, lossless summary; vague descriptions like "notes" / "misc" are unacceptable. The description alone should convey all key information.
|
||||
- **Never set `status`** — it is a field reserved for downstream processing.
|
||||
|
||||
## Tool Call Format — CRITICAL
|
||||
|
||||
You have access to tools (e.g. `daily_write`, `read`, `edit`, `write`, `frontmatter_update`). You MUST invoke them as **structured tool calls** (function calls), NOT as plain text output.
|
||||
|
||||
- **NEVER** write tool invocations as text strings such as `daily_write name=...`. Text output does NOT execute tools — no memory will be saved.
|
||||
- **ALWAYS** use the tool call mechanism provided by the runtime. The system handles parameter passing automatically.
|
||||
- If you write a tool name and its parameters as text, NO tool will be executed and NO memory will be saved.
|
||||
system_prompt_zh: |
|
||||
你是自动记忆提取系统。你的任务是从对话中提取所有有价值的信息，并记录到结构化的日记记忆中。确保**完整覆盖**——每一条值得保留的信息都必须被捕获。
|
||||
|
||||
## 提取原则
|
||||
|
||||
1. **完整覆盖**：生成足够的内容以确保对话中的所有信息都被捕获。遗漏信息比冗余更糟。
|
||||
2. **强制消歧**：绝对禁止使用代词（他、她、它、他们、这个、那个）或相对时间（昨天、今天、上周、明天、最近）。必须使用全名、明确日期和具体引用。
|
||||
3. **无损信息**：每条记录的事实必须是完整的、独立的、可理解的陈述。没有上下文的读者必须能单独理解每条记录。
|
||||
4. **原文保留**：逐字引用原始措辞、数字、URL、命令和代码片段。
|
||||
|
||||
## 提取什么
|
||||
|
||||
- **人物**：提及的所有人名（尽量使用全名）
|
||||
- **实体**：公司、产品、组织、项目、仓库
|
||||
- **决策与事件**：发生了什么、决定了什么、为什么
|
||||
- **事实与偏好**：关于用户的持久事实——身份、习惯、偏好、约束
|
||||
- **流程与方案**：可直接复用的操作步骤、命令、配置
|
||||
- **当前状态**：进度、卡点、下一步、未决问题
|
||||
- **时间信息**：当对话中明确提及时间时，使用 ISO 8601 绝对日期格式（YYYY-MM-DD）
|
||||
- **地点**：提及的具体地点名称
|
||||
- **主题关键词**：标识主题的核心关键词
|
||||
|
||||
## 正文格式
|
||||
|
||||
使用带标题的结构化分节。每条事实记录应是独立的列表项——完整且自包含。按主题分组。在末尾添加 `## Metadata` 部分，列出提取的关键词、人物、实体和主题。
|
||||
|
||||
## Frontmatter 规则
|
||||
|
||||
- `name` = 简洁、稳定的主题/事件文件名 stem，例如 `cold-remedies` 或 `project-kickoff-decision`。不要包含今天日期或日记目录日期；外层日记路径已经记录日期。对已有笔记，如果明显有更好的文件名，就更新它。
|
||||
- `description` = 详尽的无损总结；模糊的描述如 "notes" / "misc" 不可接受。仅凭 description 就应能传达全部核心信息。
|
||||
- **永远不要设置 `status`**——它是下游处理保留的字段。
|
||||
|
||||
## 工具调用格式 — 关键
|
||||
|
||||
你可以使用工具（如 `daily_write`、`read`、`edit`、`write`、`frontmatter_update`）。你**必须**以**结构化工具调用**（函数调用）的方式调用它们，而不是以纯文本输出。
|
||||
|
||||
- **绝不**将工具调用作为文本字符串输出，如 `daily_write name=...`。文本输出**不会**执行工具——不会保存任何记忆。
|
||||
- **始终**使用运行时提供的工具调用机制。系统会自动处理参数传递。
|
||||
- 如果你将工具名和参数作为文本输出，**不会有任何工具被执行**，也**不会**保存任何记忆。
|
||||
|
||||
user_message_create: |
|
||||
Today: {today}
|
||||
Extra hint: {note}
|
||||
Session ID: {session_id}
|
||||
|
||||
# Recent Conversation
|
||||
|
||||
{history}
|
||||
|
||||
# Your Task
|
||||
|
||||
Extract all valuable information from the conversation above and record them into a new daily memory note.
|
||||
|
||||
## Step 1 — Skip Check
|
||||
|
||||
Did the conversation produce substantive information worth long-term memory? Pure greetings or small talk → reply with a brief skip message and stop (do not call any tools).
|
||||
|
||||
When truly ambiguous, default to writing — losing a memory is worse than writing one extra note.
|
||||
|
||||
## Step 2 — Extract & Write
|
||||
|
||||
Analyze the conversation and extract:
|
||||
- **keywords**: Core keywords (names, places, entities, topic words)
|
||||
- **persons**: All person names mentioned
|
||||
- **entities**: Companies, products, organizations, projects
|
||||
- **topic**: The main topic(s) of the conversation
|
||||
- **timestamp**: Absolute time in ISO 8601 format (if explicit time mentioned)
|
||||
- **location**: Specific location name (if mentioned)
|
||||
|
||||
Then create the note by **calling the `daily_write` tool** with these parameters:
|
||||
- `name`: <a concise, stable topic/event filename stem in kebab-case>
|
||||
- `description`: <a thorough, lossless summary>
|
||||
- `session_id`: {session_id}
|
||||
- `date`: {today}
|
||||
- `content`: <the full body text>
|
||||
|
||||
⚠️ CRITICAL: You MUST invoke `daily_write` as a **tool call** (function call). Do NOT write `daily_write name=...` as plain text — that will NOT execute the tool and NO memory will be saved.
|
||||
|
||||
Body requirements:
|
||||
- Each factual entry must be a complete, independent, self-contained statement (lossless restatement).
|
||||
- PROHIBIT pronouns (he/she/it/they/this/that) — use full names and explicit references.
|
||||
- PROHIBIT relative time (yesterday/today/last week) — use absolute dates (YYYY-MM-DD).
|
||||
- Group entries by topic with headings.
|
||||
- End with a `## Metadata` section containing: keywords, persons, entities, topic.
|
||||
|
||||
Filename and description rules:
|
||||
- `name`: concise, stable topic/event filename stem in kebab-case. No date, no slash/backslash, no `< > : " | ? *`.
|
||||
- `description`: a thorough, lossless summary — specific enough that the description alone conveys all key information.
|
||||
|
||||
## Step 3 — Summary
|
||||
|
||||
State in one sentence what you did (which file was created). This is your final text output.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Create at most 3 notes for this session. If the conversation covers multiple distinct topics, split them into separate notes (one topic per note). If only one topic is discussed, create just one note.
|
||||
user_message_create_zh: |
|
||||
今天：{today}
|
||||
额外提示：{note}
|
||||
Session ID：{session_id}
|
||||
|
||||
# 最近的对话
|
||||
|
||||
{history}
|
||||
|
||||
# 你的任务
|
||||
|
||||
从上述对话中提取所有有价值的信息，并记录到一条新的日记记忆中。
|
||||
|
||||
## 步骤 1 — 跳过检查
|
||||
|
||||
对话是否产生了值得长期记忆的实质性信息？纯粹的寒暄或闲聊 → 回复一条简短的跳过消息并停止（不调用任何工具）。
|
||||
|
||||
当真正模棱两可时，默认写入——丢失记忆比多写一条笔记更糟。
|
||||
|
||||
## 步骤 2 — 提取并写入
|
||||
|
||||
分析对话并提取：
|
||||
- **keywords**：核心关键词（人名、地点、实体、主题词）
|
||||
- **persons**：提及的所有人名
|
||||
- **entities**：公司、产品、组织、项目
|
||||
- **topic**：对话的主要主题
|
||||
- **timestamp**：ISO 8601 格式的绝对时间（如果对话中明确提及时间）
|
||||
- **location**：具体地点名称（如果提及）
|
||||
|
||||
然后**调用 `daily_write` 工具**，使用以下参数一次性创建笔记：
|
||||
- `name`：<简洁、稳定的主题/事件文件名 stem，采用 kebab-case>
|
||||
- `description`：<详尽的无损总结>
|
||||
- `session_id`：{session_id}
|
||||
- `date`：{today}
|
||||
- `content`：<完整正文>
|
||||
|
||||
⚠️ 关键：你**必须**以**工具调用**（函数调用）方式调用 `daily_write`。不要将 `daily_write name=...` 作为纯文本输出——那样**不会**执行工具，也**不会**保存任何记忆。
|
||||
|
||||
正文要求：
|
||||
- 每条事实记录必须是完整的、独立的、自包含的陈述（无损复述）。
|
||||
- 禁止使用代词（他/她/它/他们/这个/那个）——必须使用全名和明确引用。
|
||||
- 禁止使用相对时间（昨天/今天/上周）——必须使用绝对日期（YYYY-MM-DD）。
|
||||
- 按主题分组并使用标题。
|
||||
- 末尾添加 `## Metadata` 部分，包含：keywords、persons、entities、topic。
|
||||
|
||||
文件名和描述规则：
|
||||
- `name`：简洁、稳定的主题/事件文件名 stem，采用 kebab-case。不含日期、不含 slash/反斜杠、不含 `< > : " | ? *`。
|
||||
- `description`：详尽的无损总结——具体到仅凭 description 就能传达全部核心信息。
|
||||
|
||||
## 步骤 3 — 总结
|
||||
|
||||
用一句话说明你做了什么（创建了哪个文件）。这是你最后一次文本输出。
|
||||
|
||||
## 边界
|
||||
|
||||
- 当前 session 最多创建 3 条笔记。如果对话涵盖多个不同主题，将其拆分为独立笔记（每个主题一条）。如果只讨论了一个主题，则只创建一条笔记。
|
||||
|
||||
user_message_update: |
|
||||
Today: {today}
|
||||
Extra hint: {note}
|
||||
Target path: {note_path}
|
||||
|
||||
# Recent Conversation
|
||||
|
||||
{history}
|
||||
|
||||
# Your Task
|
||||
|
||||
Extract all valuable information from the conversation above and merge them into the existing daily note at the target path.
|
||||
|
||||
## Step 1 — Skip Check
|
||||
|
||||
Did the conversation produce substantive information worth long-term memory? Pure greetings or small talk → reply with a brief skip message and stop (do not call any tools).
|
||||
|
||||
When truly ambiguous, default to writing — losing a memory is worse than writing one extra note.
|
||||
|
||||
## Step 2 — Read Existing Content
|
||||
|
||||
**Call the `read` tool** with parameter `path` set to `{note_path}` to inspect the current note content. You MUST invoke it as a tool call, not as text.
|
||||
- If the body is empty (only frontmatter, no actual content) → treat as new, jump to **Step 3b**.
|
||||
- If there is body content → go to **Step 3a** to merge.
|
||||
|
||||
## Step 3a — Merge Update
|
||||
|
||||
The note already has content. Extract new information from the conversation and merge into it.
|
||||
|
||||
Extraction rules (apply to ALL new entries):
|
||||
- Each new fact must be a complete, independent, self-contained statement (lossless restatement).
|
||||
- PROHIBIT pronouns (he/she/it/they/this/that) — use full names and explicit references.
|
||||
- PROHIBIT relative time (yesterday/today/last week) — use absolute dates (YYYY-MM-DD).
|
||||
|
||||
Merge rules:
|
||||
- **Timeline / history entries**: append only, never delete existing entries.
|
||||
- **Current-state entries** (progress, blockers, next steps, open questions): rewrite the entire section to reflect the latest snapshot.
|
||||
- **Everything else**: merge and deduplicate — keep all old facts, add new facts, remove exact duplicates.
|
||||
- **Metadata section**: merge keywords, persons, entities, topics — union of old and new values.
|
||||
|
||||
Execution:
|
||||
1. **Call the `edit` tool** for each section that needs updating, with parameters: `path`="{note_path}", `old`=<original fragment>, `new`=<replacement fragment>. You may call `edit` multiple times. Each call MUST be a tool call, not text.
|
||||
2. After body changes, **call the `frontmatter_update` tool** to refresh frontmatter, with parameters: `path`="{note_path}", `metadata`={{"name": "<updated filename stem>", "description": "<updated summary>"}}.
|
||||
- Keep the existing `name` only when it is already the best concise topic/event filename stem. The system will rename the file after your final response.
|
||||
- Do not add today's date or the daily directory date to `name`.
|
||||
3. If `edit` fails repeatedly (e.g., cannot find the original text due to formatting mismatch), fall back to **calling the `write` tool** with parameters: `path`="{note_path}", `name`=<name>, `description`=<description>, `content`=<full body> for a complete rewrite.
|
||||
|
||||
## Step 3b — Full Write (Empty File Fallback)
|
||||
|
||||
The file exists but its body is empty. Extract and write the full content by **calling the `write` tool** with parameters: `path`="{note_path}", `name`=<name>, `description`=<description>, `content`=<body>.
|
||||
|
||||
Body requirements:
|
||||
- Each factual entry must be a complete, independent, self-contained statement.
|
||||
- PROHIBIT pronouns — use full names and explicit references.
|
||||
- PROHIBIT relative time — use absolute dates (YYYY-MM-DD).
|
||||
- Group entries by topic with headings.
|
||||
- End with a `## Metadata` section containing: keywords, persons, entities, topic.
|
||||
|
||||
Filename and description rules:
|
||||
- `name`: concise, stable topic/event filename stem in kebab-case. No date.
|
||||
- `description`: a thorough, lossless summary — the description alone should convey all key information.
|
||||
|
||||
## Step 4 — Summary
|
||||
|
||||
State in one sentence what you did (what content was updated). This is your final text output.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Only operate on one target path: `{note_path}`. Do not touch other notes.
|
||||
- `write` unconditionally overwrites body and frontmatter — use with caution.
|
||||
- Filename changes are expressed by updating frontmatter `name`.
|
||||
user_message_update_zh: |
|
||||
今天：{today}
|
||||
额外提示：{note}
|
||||
目标路径：{note_path}
|
||||
|
||||
# 最近的对话
|
||||
|
||||
{history}
|
||||
|
||||
# 你的任务
|
||||
|
||||
从上述对话中提取所有有价值的信息，并合并到目标路径的已有日记中。
|
||||
|
||||
## 步骤 1 — 跳过检查
|
||||
|
||||
对话是否产生了值得长期记忆的实质性信息？纯粹的寒暄或闲聊 → 回复一条简短的跳过消息并停止（不调用任何工具）。
|
||||
|
||||
当真正模棱两可时，默认写入——丢失记忆比多写一条笔记更糟。
|
||||
|
||||
## 步骤 2 — 读取现有内容
|
||||
|
||||
**调用 `read` 工具**，参数 `path` 设为 `{note_path}`，查看当前笔记内容。必须以工具调用方式调用，不要作为文本输出。
|
||||
- 如果正文为空（只有 frontmatter 无实际内容）→ 按新建处理，跳到 **步骤 3b**。
|
||||
- 如果有正文内容 → 转到 **步骤 3a** 进行合并。
|
||||
|
||||
## 步骤 3a — 合并更新
|
||||
|
||||
笔记已有内容。从对话中提取新信息并合并进去。
|
||||
|
||||
提取规则（适用于所有新条目）：
|
||||
- 每条新事实必须是完整的、独立的、自包含的陈述（无损复述）。
|
||||
- 禁止使用代词（他/她/它/他们/这个/那个）——必须使用全名和明确引用。
|
||||
- 禁止使用相对时间（昨天/今天/上周）——必须使用绝对日期（YYYY-MM-DD）。
|
||||
|
||||
合并规则：
|
||||
- **时间线 / 历史条目**：仅追加，永远不删除已有条目。
|
||||
- **当下状态类条目**（进度、卡点、下一步、未决问题）：整段重写，反映最新快照。
|
||||
- **其余内容**：合并去重——保留全部旧事实，添加新事实，去除完全重复项。
|
||||
- **Metadata 部分**：合并 keywords、persons、entities、topics——取旧值和新值的并集。
|
||||
|
||||
执行：
|
||||
1. **调用 `edit` 工具**更新需要修改的部分，参数为：`path`="{note_path}"、`old`=<原文片段>、`new`=<替换片段>。可以多次调用 `edit`。每次调用必须是工具调用，不是文本。
|
||||
2. 正文变更后，**调用 `frontmatter_update` 工具**刷新 frontmatter，参数为：`path`="{note_path}"、`metadata`={{"name": "<更新后的文件名 stem>", "description": "<更新后的总结>"}}。
|
||||
- 只有当前 `name` 已经是最合适的简洁主题/事件文件名 stem 时才保留。系统会在你最终回复后负责重命名文件。
|
||||
- 不要在 `name` 中加入今天日期或日记目录日期。
|
||||
3. 如果 `edit` 多次失败（如因格式不匹配找不到原文），退回**调用 `write` 工具**全量重写，参数为：`path`="{note_path}"、`name`=<name>、`description`=<description>、`content`=<完整正文>。
|
||||
|
||||
## 步骤 3b — 全量写入（空文件 fallback）
|
||||
|
||||
文件存在但正文为空。提取并**调用 `write` 工具**一次性写入完整内容，参数为：`path`="{note_path}"、`name`=<name>、`description`=<description>、`content`=<正文>。
|
||||
|
||||
正文要求：
|
||||
- 每条事实记录必须是完整的、独立的、自包含的陈述。
|
||||
- 禁止使用代词——必须使用全名和明确引用。
|
||||
- 禁止使用相对时间——必须使用绝对日期（YYYY-MM-DD）。
|
||||
- 按主题分组并使用标题。
|
||||
- 末尾添加 `## Metadata` 部分，包含：keywords、persons、entities、topic。
|
||||
|
||||
文件名和描述规则：
|
||||
- `name`：简洁、稳定的主题/事件文件名 stem，采用 kebab-case。不含日期。
|
||||
- `description`：详尽的无损总结——仅凭 description 就应能传达全部核心信息。
|
||||
|
||||
## 步骤 4 — 总结
|
||||
|
||||
用一句话说明你做了什么（更新了哪些内容）。这是你最后一次文本输出。
|
||||
|
||||
## 边界
|
||||
|
||||
- 只针对一个目标路径：`{note_path}`。不要碰其他笔记。
|
||||
- `write` 会无条件覆盖正文和 frontmatter，请谨慎使用。
|
||||
- 文件名变化通过更新 frontmatter `name` 表达。
|
||||
366
reme/steps/benchmark/beam/llm_judge.py
Normal file
366
reme/steps/benchmark/beam/llm_judge.py
Normal file
|
|
@ -0,0 +1,366 @@
|
|||
"""BEAM rubric-based LLM-as-Judge step.
|
||||
|
||||
For each rubric item, calls the judge LLM with the unified judge prompt
|
||||
and collects scores (0.0 / 0.5 / 1.0). The final ``llm_judge_score`` is
|
||||
the average across all rubric items.
|
||||
|
||||
For ``event_ordering`` questions, additionally computes:
|
||||
- LLM-based event alignment (matching system events to reference events)
|
||||
- precision / recall / f1 (set-intersection after alignment)
|
||||
- Kendall's tau (ordering correlation, pure numpy implementation)
|
||||
- final_score = tau_norm * f1
|
||||
|
||||
A ``semantic`` alignment path is also available which uses ReMe's
|
||||
configured ``as_embedding`` model (replacing BEAM's sentence_transformers).
|
||||
|
||||
This replicates the evaluation logic from
|
||||
``benchmark/datasets/BEAM/src/evaluation/compute_metrics.py``.
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
from typing import List, Tuple
|
||||
|
||||
import numpy as np
|
||||
from json_repair import repair_json
|
||||
|
||||
from ...base_step import BaseStep, Ref
|
||||
from ....components import R
|
||||
from ....components.as_embedding import BaseAsEmbedding
|
||||
from ....enumeration import ComponentEnum
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# JSON parsing helper (replicates BEAM's parse_json_response)
|
||||
# ---------------------------------------------------------------------------
|
||||
def _parse_json_response(response: str) -> dict:
|
||||
response = response.strip()
|
||||
|
||||
if response.startswith("```"):
|
||||
match = re.search(
|
||||
r"```(?:json)?\s*(\[.*\]|\{.*\})\s*```",
|
||||
response,
|
||||
re.DOTALL,
|
||||
)
|
||||
if match:
|
||||
response = match.group(1).strip()
|
||||
|
||||
try:
|
||||
return json.loads(response)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
match = re.search(r"(\{.*?\}|\[.*?\])", response, re.DOTALL)
|
||||
if match:
|
||||
json_part = match.group(1)
|
||||
try:
|
||||
return json.loads(json_part)
|
||||
except Exception as e:
|
||||
raise ValueError(f"Found possible JSON but failed to parse it: {e}") from e
|
||||
|
||||
raise ValueError("No valid JSON found in response.")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Event-ordering helpers (replicate BEAM's compute_metrics.py)
|
||||
# ---------------------------------------------------------------------------
|
||||
async def _llm_equivalence(agent_wrapper, reference: str, system: str) -> bool:
|
||||
"""Binary classifier: do the two snippets describe the SAME event/fact?
|
||||
|
||||
Replicates BEAM's ``llm_equivalence`` using ``agent_wrapper.reply()``.
|
||||
"""
|
||||
system_prompt = (
|
||||
"You are a binary classifier.\n"
|
||||
"If the TWO snippets describe the SAME event/fact, reply **YES**\n"
|
||||
"Otherwise reply **NO**. No extra words.\n"
|
||||
"DO NOT provide any explanation."
|
||||
)
|
||||
user_prompt = f"First snippet: {reference}\n\nSecond snippet: {system}"
|
||||
|
||||
result = await agent_wrapper.reply(user_prompt, system_prompt=system_prompt)
|
||||
raw = (result.get("result") or "").strip().lower()
|
||||
return "yes" in raw
|
||||
|
||||
|
||||
async def _align_with_llm(
|
||||
agent_wrapper,
|
||||
reference: List[str],
|
||||
system: List[str],
|
||||
) -> Tuple[List[str], List[str]]:
|
||||
"""Align system events to reference events via LLM equivalence.
|
||||
|
||||
Replicates BEAM's ``align_with_llm``: for each system event, find the
|
||||
first unmatched reference event that is LLM-equivalent. If found,
|
||||
replace the system event with the reference text (canonicalisation).
|
||||
Ensures 1-to-1 mapping.
|
||||
"""
|
||||
used = set()
|
||||
system_out = []
|
||||
|
||||
for s in system:
|
||||
matched_index = None
|
||||
for index, r in enumerate(reference):
|
||||
if index in used:
|
||||
continue
|
||||
if await _llm_equivalence(agent_wrapper, reference=r, system=s):
|
||||
matched_index = index
|
||||
break
|
||||
|
||||
if matched_index is not None:
|
||||
system_out.append(reference[matched_index])
|
||||
used.add(matched_index)
|
||||
else:
|
||||
system_out.append(s)
|
||||
|
||||
return reference, system_out
|
||||
|
||||
|
||||
async def _semantic_align(
|
||||
embedding_fn,
|
||||
reference: List[str],
|
||||
system: List[str],
|
||||
thr: float = 0.65,
|
||||
) -> Tuple[List[str], List[str]]:
|
||||
"""Align system events to reference events via embedding cosine similarity.
|
||||
|
||||
Replaces BEAM's ``semantic_align`` (which used sentence_transformers)
|
||||
with ReMe's configured ``as_embedding`` model.
|
||||
"""
|
||||
if not reference or not system:
|
||||
return reference, system
|
||||
|
||||
ref_embeddings = np.array(await embedding_fn(reference))
|
||||
sys_embeddings = np.array(await embedding_fn(system))
|
||||
|
||||
# Normalise
|
||||
ref_norms = ref_embeddings / (np.linalg.norm(ref_embeddings, axis=1, keepdims=True) + 1e-12)
|
||||
sys_norms = sys_embeddings / (np.linalg.norm(sys_embeddings, axis=1, keepdims=True) + 1e-12)
|
||||
|
||||
used_reference = set()
|
||||
system_canon = []
|
||||
|
||||
for i, s_txt in enumerate(system):
|
||||
sims = sys_norms[i] @ ref_norms.T # cosine similarity
|
||||
best = int(np.argmax(sims))
|
||||
if sims[best] >= thr and best not in used_reference:
|
||||
system_canon.append(reference[best])
|
||||
used_reference.add(best)
|
||||
else:
|
||||
system_canon.append(s_txt)
|
||||
|
||||
return reference, system_canon
|
||||
|
||||
|
||||
def _kendall_tau_b(x: list, y: list) -> float:
|
||||
"""Compute Kendall's tau-b rank correlation using only numpy.
|
||||
|
||||
Replicates ``scipy.stats.kendalltau(x, y, variant='b')`` for the
|
||||
rank-based inputs used in event ordering scoring.
|
||||
"""
|
||||
x_arr = np.asarray(x, dtype=float)
|
||||
y_arr = np.asarray(y, dtype=float)
|
||||
n = len(x_arr)
|
||||
if n < 2:
|
||||
return 0.0
|
||||
|
||||
concordant = 0
|
||||
discordant = 0
|
||||
x_ties = 0
|
||||
y_ties = 0
|
||||
|
||||
for i in range(n - 1):
|
||||
for j in range(i + 1, n):
|
||||
dx = x_arr[j] - x_arr[i]
|
||||
dy = y_arr[j] - y_arr[i]
|
||||
if dx == 0 and dy == 0:
|
||||
x_ties += 1
|
||||
y_ties += 1
|
||||
elif dx == 0:
|
||||
x_ties += 1
|
||||
elif dy == 0:
|
||||
y_ties += 1
|
||||
elif (dx > 0) == (dy > 0):
|
||||
concordant += 1
|
||||
else:
|
||||
discordant += 1
|
||||
|
||||
n0 = n * (n - 1) / 2
|
||||
denom = np.sqrt((n0 - x_ties) * (n0 - y_ties))
|
||||
if denom == 0:
|
||||
return 0.0
|
||||
return (concordant - discordant) / denom
|
||||
|
||||
|
||||
def _event_ordering_score(
|
||||
reference_canon: List[str],
|
||||
system_canon: List[str],
|
||||
) -> dict:
|
||||
"""Compute precision/recall/f1 + Kendall's tau after alignment.
|
||||
|
||||
Replicates BEAM's ``event_ordering_score`` (the scoring part, after
|
||||
alignment is done).
|
||||
"""
|
||||
tp = len(set(reference_canon) & set(system_canon))
|
||||
fp = len([x for x in system_canon if x not in reference_canon])
|
||||
fn = len([x for x in reference_canon if x not in system_canon])
|
||||
|
||||
precision = tp / (tp + fp) if tp + fp else 0
|
||||
recall = tp / (tp + fn) if tp + fn else 0
|
||||
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) else 0
|
||||
|
||||
union = list(dict.fromkeys(reference_canon + system_canon))
|
||||
tie_rank = len(union) + 1
|
||||
|
||||
def to_rank(seq):
|
||||
r = {item: i + 1 for i, item in enumerate(seq)}
|
||||
return [r.get(u, tie_rank) for u in union]
|
||||
|
||||
tau_b = _kendall_tau_b(
|
||||
to_rank(reference_canon),
|
||||
to_rank(system_canon),
|
||||
)
|
||||
tau_b_norm = (tau_b + 1) / 2 if tau_b is not None else 0
|
||||
|
||||
final_score = tau_b_norm * f1
|
||||
return {
|
||||
"precision": precision,
|
||||
"recall": recall,
|
||||
"f1": f1,
|
||||
"tau_norm": tau_b_norm,
|
||||
"final_score": final_score,
|
||||
}
|
||||
|
||||
|
||||
@R.register("beam_rubric_judge_step")
|
||||
class BeamRubricJudgeStep(BaseStep):
|
||||
"""Judge an LLM response against a list of rubric criteria.
|
||||
|
||||
Inputs (from RuntimeContext):
|
||||
llm_response (str, required): The model's response to evaluate.
|
||||
rubric (list[str], required): Rubric criteria to check.
|
||||
probing_question (str, optional): The original probing question.
|
||||
question_type (str, optional): Question type (e.g. "event_ordering").
|
||||
|
||||
Output (written to context.response):
|
||||
answer = str(llm_judge_score)
|
||||
metadata["llm_judge_score"] = float
|
||||
metadata["llm_judge_responses"] = list[dict]
|
||||
metadata["event_ordering"] = dict (only for event_ordering type)
|
||||
"""
|
||||
|
||||
as_embedding: BaseAsEmbedding = Ref(
|
||||
BaseAsEmbedding,
|
||||
ComponentEnum.AS_EMBEDDING,
|
||||
optional=True,
|
||||
)
|
||||
|
||||
async def execute(self):
|
||||
assert self.context is not None
|
||||
llm_response: str = self.context.get("llm_response", "")
|
||||
rubric: list[str] = self.context.get("rubric", [])
|
||||
probing_question: str = self.context.get("probing_question", "")
|
||||
question_type: str = self.context.get("question_type", "")
|
||||
|
||||
if not llm_response:
|
||||
raise ValueError("beam_rubric_judge_step requires non-empty llm_response")
|
||||
if not rubric:
|
||||
raise ValueError("beam_rubric_judge_step requires non-empty rubric")
|
||||
if self.agent_wrapper is None:
|
||||
raise RuntimeError("beam_rubric_judge_step requires agent_wrapper")
|
||||
|
||||
# ----- Standard rubric-based LLM-as-Judge (all question types) -----
|
||||
judge_template = self.get_prompt("judge_prompt")
|
||||
|
||||
llm_judge_responses: list[dict] = []
|
||||
total_score = 0.0
|
||||
|
||||
for item in rubric:
|
||||
prompt = judge_template.replace("<rubric_item>", item).replace("<llm_response>", llm_response)
|
||||
|
||||
result = await self.agent_wrapper.reply(prompt)
|
||||
raw = (result.get("result") or "").strip()
|
||||
|
||||
try:
|
||||
parsed = _parse_json_response(raw)
|
||||
except Exception:
|
||||
try:
|
||||
parsed = json.loads(repair_json(raw))
|
||||
except Exception:
|
||||
parsed = {"score": 0.0, "reason": f"Failed to parse: {raw[:200]}"}
|
||||
|
||||
score = float(parsed.get("score", 0))
|
||||
|
||||
# Abstention: binary classification — 1.0 stays 1, <1.0 becomes 0
|
||||
if question_type == "abstention":
|
||||
score = 1.0 if score >= 1.0 else 0.0
|
||||
|
||||
total_score += score
|
||||
llm_judge_responses.append(parsed)
|
||||
|
||||
llm_judge_score = total_score / len(rubric) if rubric else 0.0
|
||||
|
||||
self.logger.info(f"[{self.name}] judge score: {llm_judge_score:.3f}")
|
||||
|
||||
self.context.response.success = True
|
||||
self.context.response.answer = str(llm_judge_score)
|
||||
self.context.response.metadata.update(
|
||||
{
|
||||
"llm_judge_score": llm_judge_score,
|
||||
"llm_judge_responses": llm_judge_responses,
|
||||
"rubric": rubric,
|
||||
"llm_response": llm_response,
|
||||
"probing_question": probing_question,
|
||||
"question_type": question_type,
|
||||
},
|
||||
)
|
||||
|
||||
# ----- event_ordering extra metrics -----
|
||||
# Replicates BEAM's evaluate_event_ordering: system_list = llm_response.split("\n")
|
||||
# Note: BEAM calls extract_facts first but immediately overwrites with split("\n").
|
||||
if question_type == "event_ordering":
|
||||
eo_metrics = await self._compute_event_ordering(
|
||||
rubric=rubric,
|
||||
llm_response=llm_response,
|
||||
)
|
||||
self.context.response.metadata["event_ordering"] = eo_metrics
|
||||
self.logger.info(f"[{self.name}] event_ordering: {eo_metrics}")
|
||||
|
||||
return self.context.response
|
||||
|
||||
async def _compute_event_ordering(
|
||||
self,
|
||||
rubric: list[str],
|
||||
llm_response: str,
|
||||
) -> dict:
|
||||
"""Compute event_ordering extra metrics.
|
||||
|
||||
Uses ``align_type="llm"`` to match BEAM's original code.
|
||||
Also supports ``align_type="semantic"`` via ReMe's embedding model
|
||||
(replacing sentence_transformers).
|
||||
"""
|
||||
# BEAM: system_list = llm_response.split("\n")
|
||||
system_list = [line for line in llm_response.split("\n") if line.strip()]
|
||||
|
||||
# Use LLM alignment (matching BEAM's align_type="llm")
|
||||
reference_canon, system_canon = await _align_with_llm(
|
||||
agent_wrapper=self.agent_wrapper,
|
||||
reference=rubric,
|
||||
system=system_list,
|
||||
)
|
||||
|
||||
eo_score = _event_ordering_score(reference_canon, system_canon)
|
||||
|
||||
# Also compute semantic alignment if embedding is available
|
||||
if self.as_embedding is not None:
|
||||
try:
|
||||
ref_canon_sem, sys_canon_sem = await _semantic_align(
|
||||
embedding_fn=self.as_embedding,
|
||||
reference=rubric,
|
||||
system=system_list,
|
||||
)
|
||||
eo_score_sem = _event_ordering_score(ref_canon_sem, sys_canon_sem)
|
||||
eo_score["semantic_alignment"] = eo_score_sem
|
||||
except Exception as e:
|
||||
self.logger.warning(f"[{self.name}] semantic_align failed: {e}")
|
||||
|
||||
return eo_score
|
||||
70
reme/steps/benchmark/beam/llm_judge.yaml
Normal file
70
reme/steps/benchmark/beam/llm_judge.yaml
Normal file
|
|
@ -0,0 +1,70 @@
|
|||
judge_prompt: |
|
||||
You are an expert evaluator tasked with judging whether the LLM's response demonstrates compliance with the specified RUBRIC CRITERION.
|
||||
|
||||
## EVALUATION INPUTS
|
||||
- QUESTION (what the user asked): <question>
|
||||
- RUBRIC CRITERION (what to check): <rubric_item>
|
||||
- RESPONSE TO EVALUATE: <llm_response>
|
||||
|
||||
## EVALUATION RUBRIC:
|
||||
The rubric defines a specific requirement, constraint, or expected behavior that the LLM response should demonstrate.
|
||||
|
||||
**IMPORTANT**: Pay careful attention to whether the rubric specifies:
|
||||
- **Positive requirements** (things the response SHOULD include/do)
|
||||
- **Negative constraints** (things the response SHOULD NOT include/do, often indicated by "no", "not", "avoid", "absent")
|
||||
|
||||
## RESPONSIVENESS REQUIREMENT (anchored to the QUESTION)
|
||||
A compliant response must be **on-topic with respect to the QUESTION** and attempt to answer it.
|
||||
- If the response does not address the QUESTION, score **0.0** and stop.
|
||||
- For negative constraints, both must hold: (a) the response is responsive to the QUESTION, and (b) the prohibited element is absent.
|
||||
|
||||
## SEMANTIC TOLERANCE RULES:
|
||||
Judge by meaning, not exact wording.
|
||||
- Accept **paraphrases** and **synonyms** that preserve intent.
|
||||
- **Case/punctuation/whitespace** differences must be ignored.
|
||||
- **Numbers/currencies/dates** may appear in equivalent forms (e.g., "$68,000", "68k", "68,000 USD", or "sixty-eight thousand dollars"). Treat them as equal when numerically equivalent.
|
||||
- If the rubric expects a number or duration, prefer **normalized comparison** (extract and compare values) over string matching.
|
||||
|
||||
## STYLE NEUTRALITY (prevents style contamination):
|
||||
Ignore tone, politeness, length, and flourish unless the rubric explicitly requires a format/structure (e.g., "itemized list", "no citations", "one sentence").
|
||||
- Do **not** penalize hedging, voice, or verbosity if content satisfies the rubric.
|
||||
- Only evaluate format when the rubric **explicitly** mandates it.
|
||||
|
||||
## SCORING SCALE:
|
||||
- **1.0 (Complete Compliance)**: Fully complies with the rubric criterion.
|
||||
- Positive: required element present, accurate, properly executed (allowing semantic equivalents).
|
||||
- Negative: prohibited element **absent** AND response is **responsive**.
|
||||
|
||||
- **0.5 (Partial Compliance)**: Partially complies.
|
||||
- Positive: element present but minor inaccuracies/incomplete execution.
|
||||
- Negative: generally responsive and mostly avoids the prohibited element but with minor/edge violations.
|
||||
|
||||
- **0.0 (No Compliance)**: Fails to comply.
|
||||
- Positive: required element missing or incorrect.
|
||||
- Negative: prohibited element present **or** response is non-responsive/evasive even if the element is absent.
|
||||
|
||||
## EVALUATION INSTRUCTIONS:
|
||||
1. **Understand the Requirement**: Determine if the rubric is asking for something to be present (positive) or absent (negative/constraint).
|
||||
|
||||
2. **Parse Compound Statements**: If the rubric contains multiple elements connected by "and" or commas, evaluate whether:
|
||||
- **All elements** must be present for full compliance (1.0)
|
||||
- **Some elements** present indicates partial compliance (0.5)
|
||||
- **No elements** present indicates no compliance (0.0)
|
||||
|
||||
3. **Check Compliance**:
|
||||
- For positive requirements: Look for the presence and quality of the required element
|
||||
- For negative constraints: Look for the absence of the prohibited element
|
||||
|
||||
4. **Assign Score**: Based on compliance with the specific rubric criterion according to the scoring scale above.
|
||||
|
||||
5. **Provide Reasoning**: Explain whether the rubric criterion was satisfied and justify the score.
|
||||
|
||||
## OUTPUT FORMAT:
|
||||
Return your evaluation in JSON format with two fields:
|
||||
|
||||
{
|
||||
"score": [your score: 1.0, 0.5, or 0.0],
|
||||
"reason": "[detailed explanation of whether the rubric criterion was satisfied and why this justified the assigned score]"
|
||||
}
|
||||
|
||||
NOTE: ONLY output the json object, without any explanation before or after that
|
||||
|
|
@ -1,21 +1,11 @@
|
|||
"""LongMemEval benchmark steps."""
|
||||
|
||||
from .agentic_answer import LmeAgenticAnswerStep
|
||||
from .llm_judge import LmeAnswerJudgeStep
|
||||
from .auto_memory import LmeAutoMemoryStep
|
||||
from .context_answer import ContextAnswerStep
|
||||
from .extract_session import LmeExtractSessionStep
|
||||
from .final_answer_review import FinalAnswerReviewStep
|
||||
from .golden_check import GoldenCheckStep
|
||||
from .lme_llm_judge import LmeLlmJudgeStep
|
||||
from .session_review import SessionReviewStep
|
||||
|
||||
__all__ = [
|
||||
"ContextAnswerStep",
|
||||
"FinalAnswerReviewStep",
|
||||
"GoldenCheckStep",
|
||||
"LmeAgenticAnswerStep",
|
||||
"LmeAnswerJudgeStep",
|
||||
"LmeAutoMemoryStep",
|
||||
"LmeExtractSessionStep",
|
||||
"LmeLlmJudgeStep",
|
||||
"SessionReviewStep",
|
||||
]
|
||||
|
|
|
|||
|
|
@ -1,87 +1,16 @@
|
|||
"""agentic_answer — answer the LongMemEval question from the indexed memory.
|
||||
"""LongMemEval agentic answer step – ReAct agent that answers questions using the search tool."""
|
||||
|
||||
Job #4 of the pipeline. Reads ``query.json`` and hands the question to an agent
|
||||
equipped with ``vector_search`` / ``bm25_search`` / ``python_execute`` /
|
||||
``extract_session_by_id``. The agent searches the daily-note index, pivots to
|
||||
raw sessions by ``session_id`` when a hit is promising, and keeps trying until it
|
||||
can answer or has searched too many times. The final answer is written to
|
||||
``mem_answer.json`` in the workspace.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
from ...base_step import BaseStep
|
||||
from ....components import R
|
||||
from ..base import BaseAgenticAnswerStep
|
||||
|
||||
|
||||
@R.register("lme_agentic_answer_step")
|
||||
class LmeAgenticAnswerStep(BaseStep):
|
||||
"""Drive the tool-using agent that answers from indexed memory."""
|
||||
class LmeAgenticAnswerStep(BaseAgenticAnswerStep):
|
||||
"""Answer a LongMemEval query via ReAct agent with access to the search tool.
|
||||
|
||||
_OUTPUT_FILE = "mem_answer.json"
|
||||
The agent uses the ``agent_wrapper`` component in ReAct mode, calling the
|
||||
``search`` job tool to retrieve relevant memory chunks before generating
|
||||
a final answer.
|
||||
"""
|
||||
|
||||
def _load_query(self) -> dict:
|
||||
path = self.workspace_path / "query.json"
|
||||
with path.open(encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError("query.json is not a JSON object")
|
||||
return data
|
||||
|
||||
async def execute(self):
|
||||
assert self.context is not None
|
||||
if self.agent_wrapper is None:
|
||||
raise ValueError("lme_agentic_answer_step requires agent_wrapper")
|
||||
|
||||
query = self._load_query()
|
||||
question = str(query.get("question", "") or "").strip()
|
||||
question_date = str(query.get("question_date", "") or "").strip()
|
||||
question_id = str(query.get("question_id", "") or "").strip()
|
||||
if not question:
|
||||
raise ValueError("query.json requires a non-empty 'question'")
|
||||
|
||||
user_prompt = self.prompt_format(
|
||||
"user_message",
|
||||
question=question,
|
||||
question_date=question_date or "(unknown)",
|
||||
)
|
||||
# A stable tool_context_id makes vector/bm25 dedup across this answer run,
|
||||
# so repeated searches surface genuinely new chunks each time.
|
||||
result = await self.agent_wrapper.reply(
|
||||
user_prompt,
|
||||
system_prompt=self.get_prompt("system_prompt"),
|
||||
tool_context_id=question_id or question,
|
||||
)
|
||||
answer = (result.get("result") or "").strip()
|
||||
# session_id names the trajectory file mem_session/agentscope/<session_id>.jsonl,
|
||||
# so downstream tooling can locate this run's full tool-call trail.
|
||||
session_id = str(result.get("session_id") or "")
|
||||
|
||||
out_path = self.workspace_path / self._OUTPUT_FILE
|
||||
out_path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"question_id": question_id,
|
||||
"question": question,
|
||||
"answer": answer,
|
||||
"session_id": session_id,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
self.logger.info(f"[{self.name}] answer for {question_id or question!r}: {answer!r}")
|
||||
self.context.response.success = True
|
||||
self.context.response.answer = answer
|
||||
self.context.response.metadata.update(
|
||||
{
|
||||
"question_id": question_id,
|
||||
"question": question,
|
||||
"answer": answer,
|
||||
"session_id": session_id,
|
||||
"path": self._OUTPUT_FILE,
|
||||
},
|
||||
)
|
||||
return self.context.response
|
||||
TOOL_CONTEXT_PREFIX = "lme_agentic_answer"
|
||||
|
|
|
|||
|
|
@ -1,40 +1,18 @@
|
|||
system_prompt: |
|
||||
You answer a user's question using ONLY their long-term memory, retrieved through tools. The
|
||||
memory is a set of daily notes, each distilled from one past chat session and tagged in its
|
||||
header with a `session_id` pointing back to the raw session.
|
||||
You are a memory retrieval assistant. You MUST use the search tool to find information before answering.
|
||||
- Your total time of tool calls should be at most 9 times
|
||||
|
||||
Available tools:
|
||||
- `vector_search(query)`: dense semantic search over the memory notes.
|
||||
- `bm25_search(query)`: keyword search over the memory notes.
|
||||
- `extract_session_by_id(session_id)`: go back to the ORIGINAL raw session behind a note and
|
||||
pull out its full content relevant to the question. Use the `session_id` shown in a search
|
||||
result's header.
|
||||
- `python_execute(code)`: run Python for any counting, date math, or reasoning over what you found.
|
||||
## Search Strategy
|
||||
- You can call 'search' tool to search multiple times (at least once) with different queries to gather comprehensive information.
|
||||
- Do not stop after the first relevant one unless you are confident that the information is sufficient.
|
||||
## Draft Tool
|
||||
- Use 'add_draft' to save key findings during search, and 'read_all_draft' to review all saved notes before answering.
|
||||
## Answer Rules
|
||||
- Answer based ONLY on retrieved context.
|
||||
- If the memories contain contradictory information, integrate the context, query, and most importantly the timestamp of each memory to comprehensively determine the final answer.
|
||||
- Output ONLY the direct factual answer — no reasoning, no search process, no elaboration.
|
||||
- Do NOT invent or assume information that isn't in the memories.
|
||||
- If information is not founded or not sufficient after multiple searches, reply: 'Information not found.'
|
||||
|
||||
Strategy:
|
||||
1. Start by searching with the user's ORIGINAL question wording — call BOTH `vector_search` and
|
||||
`bm25_search` with it.
|
||||
2. If the results already fully support an answer, answer.
|
||||
3. If a result looks relevant but the distilled note is not enough (missing a number, exact date,
|
||||
or wording), take its `session_id` and call `extract_session_by_id` to read the raw session.
|
||||
4. Do NOT give up early. Keep trying: reformulate with new keywords, aliases, entities, dates, and
|
||||
short phrases from the question or from earlier results; search again; and call
|
||||
`extract_session_by_id` on any additional relevant sessions.
|
||||
5. Search results are deduplicated within this run, so a search returning nothing new means those
|
||||
chunks were already seen — change your wording rather than repeating it.
|
||||
6. Only after you have made MORE THAN 10 search attempts and still cannot find support, answer
|
||||
exactly: not provided
|
||||
|
||||
Answer rules:
|
||||
- Answer strictly from retrieved memory; never invent facts.
|
||||
- Be direct and specific; include the exact value/date the question asks for.
|
||||
- Your final message is the answer itself (no tool calls, no preamble).
|
||||
|
||||
user_message: |
|
||||
Question date: {question_date}
|
||||
Question: {question}
|
||||
|
||||
Find the answer in the user's long-term memory using the tools, following the strategy above.
|
||||
Remember: search with the original question first (both vector and bm25), pivot to
|
||||
`extract_session_by_id` for promising sessions, keep trying with new wording, and only answer
|
||||
"not provided" after more than 10 search attempts have failed.
|
||||
temporal_hint: |
|
||||
Current time context: {query_time}
|
||||
|
|
|
|||
|
|
@ -1,428 +1,132 @@
|
|||
"""lme_auto_memory — turn every LongMemEval session into a search-friendly note.
|
||||
"""auto_memory (lme) — AutoMemoryStep with timestamp interpolation and daily_write date default."""
|
||||
|
||||
For a workspace such as ``datasets/longmemeval/1`` this step walks each raw
|
||||
session under ``resource_dir`` (files named ``<date>_(...)_<time>@<session_id>.json``
|
||||
with ``haystack_date`` / ``haystack_session_id`` / ``messages``) and, one per
|
||||
session, asks an agent to *completely* extract its content — entities, times,
|
||||
numbers, preferences, events, causal links — into a daily note optimized for
|
||||
both BM25 and vector retrieval.
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
Each note is written to ``<daily_dir>/<YYYY-MM-DD>/<name>.md`` via the shared
|
||||
``daily_write`` job, so the frontmatter carries ``session_id`` for progressive
|
||||
expansion (the agentic-answer flow pivots from a search hit back to the raw
|
||||
session through this id). Filenames are LLM-generated topic stems; same-day
|
||||
collisions are disambiguated by appending the session id.
|
||||
"""
|
||||
from agentscope.message import Msg
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
import frontmatter
|
||||
|
||||
from ...base_step import BaseStep
|
||||
from ...file_io import extract_daily_date
|
||||
from ...evolve.auto_memory import AutoMemoryStep, _normalize_msg_timestamp
|
||||
from ....components import R
|
||||
|
||||
START_INTERVAL_SECONDS = 1.0
|
||||
MAX_CONCURRENCY = 60
|
||||
RETRY_INITIAL_SECONDS = 5.0
|
||||
RETRY_MAX_SECONDS = 300.0
|
||||
_LME_DATETIME_RE = re.compile(r"(\d{4})/(\d{2})/(\d{2}).*?(\d{2}):(\d{2})")
|
||||
_NON_RETRYABLE_DATA_INSPECTION_MARKERS = (
|
||||
"data_inspection_failed",
|
||||
"DataInspectionFailed",
|
||||
"Input text data may contain inappropriate content",
|
||||
)
|
||||
|
||||
# Structured extraction the memory agent must return per session.
|
||||
_MEMORY_SCHEMA = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "Concise, stable topic/event filename stem (kebab-case, no date, no slash or "
|
||||
"reserved characters). E.g. 'daily-commute-details' or 'leather-boot-care'.",
|
||||
},
|
||||
"description": {
|
||||
"type": "string",
|
||||
"description": "Thorough one-paragraph summary of the note body — specific enough that this "
|
||||
"description alone conveys all key facts. Used as a search-friendly abstract.",
|
||||
},
|
||||
"body": {
|
||||
"type": "string",
|
||||
"description": "Complete markdown extraction of every core fact in the session, written for "
|
||||
"retrieval (natural-language statements, explicit entities, dates and numbers verbatim).",
|
||||
},
|
||||
},
|
||||
"required": ["name", "description", "body"],
|
||||
}
|
||||
def _parse_iso_seconds(value: str) -> datetime | None:
|
||||
"""Parse an ISO-8601 timestamp that is precise to at least seconds.
|
||||
|
||||
Accepts formats like:
|
||||
2026-07-01T14:30:00
|
||||
2026-07-01T14:30:00Z
|
||||
2026-07-01T14:30:00+08:00
|
||||
2026-07-01T14:30:00.123456
|
||||
|
||||
Rejects date-only (``2026-07-01``) or minute-only (``2026-07-01T14:30``).
|
||||
Returns ``None`` when the value does not satisfy the requirements.
|
||||
"""
|
||||
text = str(value).strip()
|
||||
# Minimum valid: YYYY-MM-DDTHH:MM:SS = 19 chars
|
||||
if len(text) < 19:
|
||||
return None
|
||||
# Must contain 'T' separator and at least HH:MM:SS after it
|
||||
if "T" not in text:
|
||||
return None
|
||||
time_part = text.split("T", 1)[1]
|
||||
# time_part must start with HH:MM:SS (8 chars minimum)
|
||||
if len(time_part) < 8 or time_part[2] != ":" or time_part[5] != ":":
|
||||
return None
|
||||
try:
|
||||
return datetime.fromisoformat(text)
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def _interpolate_timestamps(items: list[dict]) -> list[dict]:
|
||||
"""Fill missing ``created_at`` fields via linear interpolation.
|
||||
|
||||
Rules (backward-compatible — returns *items* unchanged when no message
|
||||
carries a ``created_at`` value):
|
||||
|
||||
1. If **no** message has ``created_at`` → return as-is (system time used
|
||||
later by AgentScope's ``Msg`` constructor).
|
||||
2. Messages **before** the first timestamped message → inherit the first
|
||||
timestamp.
|
||||
3. Messages **after** the last timestamped message → inherit the last
|
||||
timestamp.
|
||||
4. Messages **between** two timestamped anchors → linearly interpolated.
|
||||
"""
|
||||
# Pass 1: normalize aliases and collect anchors
|
||||
normalized: list[dict] = []
|
||||
anchors: list[tuple[int, datetime]] = [] # (index, parsed_dt)
|
||||
|
||||
for i, raw_item in enumerate(items):
|
||||
if not isinstance(raw_item, dict):
|
||||
normalized.append(raw_item)
|
||||
continue
|
||||
item = _normalize_msg_timestamp(raw_item)
|
||||
normalized.append(item)
|
||||
ca = item.get("created_at")
|
||||
if ca:
|
||||
dt = _parse_iso_seconds(str(ca))
|
||||
if dt is not None:
|
||||
anchors.append((i, dt))
|
||||
|
||||
# No anchors → fully backward-compatible, let Msg use system time
|
||||
if not anchors:
|
||||
return normalized
|
||||
|
||||
# Pass 2: interpolate
|
||||
result: list[dict] = []
|
||||
for i, item in enumerate(normalized):
|
||||
if not isinstance(item, dict):
|
||||
result.append(item)
|
||||
continue
|
||||
# Already has a valid parsed anchor — keep it
|
||||
if any(idx == i for idx, _ in anchors):
|
||||
result.append(item)
|
||||
continue
|
||||
|
||||
# Find the nearest preceding and following anchors
|
||||
prev_anchor: tuple[int, datetime] | None = None
|
||||
next_anchor: tuple[int, datetime] | None = None
|
||||
for idx, dt in anchors:
|
||||
if idx < i:
|
||||
prev_anchor = (idx, dt)
|
||||
for idx, dt in anchors:
|
||||
if idx > i:
|
||||
next_anchor = (idx, dt)
|
||||
break
|
||||
|
||||
# Determine interpolated time
|
||||
if prev_anchor is None:
|
||||
# Before the first anchor
|
||||
interpolated_dt = anchors[0][1]
|
||||
elif next_anchor is None:
|
||||
# After the last anchor
|
||||
interpolated_dt = anchors[-1][1]
|
||||
else:
|
||||
# Between two anchors — linear
|
||||
prev_idx, prev_dt = prev_anchor
|
||||
next_idx, next_dt = next_anchor
|
||||
span = next_idx - prev_idx
|
||||
ratio = (i - prev_idx) / span
|
||||
delta_seconds = (next_dt - prev_dt).total_seconds()
|
||||
interpolated_dt = prev_dt + timedelta(seconds=delta_seconds * ratio)
|
||||
|
||||
item = {**item, "created_at": interpolated_dt.isoformat()}
|
||||
result.append(item)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
@R.register("lme_auto_memory_step")
|
||||
class LmeAutoMemoryStep(BaseStep):
|
||||
"""Extract each LME session into a daily note via a per-session agent."""
|
||||
class LmeAutoMemoryStep(AutoMemoryStep):
|
||||
"""AutoMemoryStep variant that interpolates timestamps and pins daily_write to the resolved day."""
|
||||
|
||||
def __init__(self, **kwargs):
|
||||
super().__init__(**kwargs)
|
||||
self._reserve_lock = asyncio.Lock()
|
||||
self._reserved: dict[tuple[str, str], str] = {}
|
||||
|
||||
def _resource_dir_name(self) -> str:
|
||||
return self.app_context.app_config.resource_dir if self.app_context is not None else "session"
|
||||
|
||||
def _session_dir(self) -> Path:
|
||||
return self.workspace_path / self._resource_dir_name()
|
||||
|
||||
@staticmethod
|
||||
def _parse_lme_datetime(raw_date: str) -> datetime | None:
|
||||
"""Parse LongMemEval timestamps like ``2023/05/20 (Sat) 03:29``."""
|
||||
match = _LME_DATETIME_RE.search(raw_date.strip())
|
||||
if match is None:
|
||||
return None
|
||||
try:
|
||||
year, month, day, hour, minute = (int(part) for part in match.groups())
|
||||
return datetime(year, month, day, hour, minute)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _parse_day(raw_date: str) -> str | None:
|
||||
"""Parse a LongMemEval ``haystack_date`` (e.g. '2023/05/20 (Sat) 03:29') to YYYY-MM-DD."""
|
||||
head = raw_date.strip()[:10].replace("/", "-")
|
||||
return extract_daily_date(head)
|
||||
|
||||
@staticmethod
|
||||
def _load_json(path: Path) -> dict:
|
||||
with path.open(encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError("session file is not a JSON object")
|
||||
return data
|
||||
|
||||
@staticmethod
|
||||
def _format_messages(messages: list) -> str:
|
||||
lines: list[str] = []
|
||||
for msg in messages:
|
||||
if not isinstance(msg, dict):
|
||||
continue
|
||||
role = str(msg.get("role", "")).strip() or "unknown"
|
||||
content = msg.get("content", "")
|
||||
if not isinstance(content, str):
|
||||
content = json.dumps(content, ensure_ascii=False)
|
||||
lines.append(f"[{role}]\n{content}")
|
||||
return "\n\n".join(lines)
|
||||
|
||||
@staticmethod
|
||||
def _is_data_inspection_error(exc: Exception) -> bool:
|
||||
text = str(exc)
|
||||
return any(marker in text for marker in _NON_RETRYABLE_DATA_INSPECTION_MARKERS)
|
||||
|
||||
async def _existing_session_id(self, rel_path: str) -> str:
|
||||
note = self.workspace_path / rel_path
|
||||
if not note.is_file():
|
||||
return ""
|
||||
try:
|
||||
post = frontmatter.loads(note.read_text(encoding="utf-8"))
|
||||
except Exception:
|
||||
return ""
|
||||
return str((post.metadata or {}).get("session_id", "") or "").strip()
|
||||
|
||||
async def _reserve_name(self, daily_dir: str, day: str, name: str, session_id: str) -> str:
|
||||
"""Pick a collision-free filename stem for this session under ``day``."""
|
||||
async with self._reserve_lock:
|
||||
for cand in (name, f"{name}-{session_id}"):
|
||||
key = (day, cand)
|
||||
owner = self._reserved.get(key)
|
||||
if owner == session_id:
|
||||
return cand
|
||||
if owner is not None:
|
||||
continue
|
||||
existing = await self._existing_session_id(f"{daily_dir}/{day}/{cand}.md")
|
||||
if existing and existing != session_id:
|
||||
continue
|
||||
self._reserved[key] = session_id
|
||||
return cand
|
||||
# Extremely unlikely fallback (same topic AND same session id twice).
|
||||
i = 2
|
||||
while True:
|
||||
cand = f"{name}-{session_id}-{i}"
|
||||
key = (day, cand)
|
||||
if key not in self._reserved:
|
||||
self._reserved[key] = session_id
|
||||
return cand
|
||||
i += 1
|
||||
|
||||
# pylint: disable=too-many-statements
|
||||
async def execute(self):
|
||||
assert self.context is not None
|
||||
if self.agent_wrapper is None:
|
||||
raise ValueError("lme_auto_memory_step requires agent_wrapper")
|
||||
|
||||
query_data = self._load_json(self.workspace_path / "query.json")
|
||||
question_date = str(query_data.get("question_date") or "").strip()
|
||||
question_dt = self._parse_lme_datetime(question_date)
|
||||
if question_dt is None:
|
||||
raise ValueError(f"query.json has an invalid 'question_date': {question_date!r}")
|
||||
|
||||
session_dir = self._session_dir()
|
||||
if not session_dir.is_dir():
|
||||
raise FileNotFoundError(f"Session directory not found: {session_dir}")
|
||||
session_files = sorted(p for p in session_dir.iterdir() if p.suffix == ".json")
|
||||
sessions: list[tuple[dict, Path, str, str, str]] = []
|
||||
filtered_sessions: list[dict] = []
|
||||
session_ids_illegal: list[str] = []
|
||||
|
||||
for session_path in session_files:
|
||||
try:
|
||||
session = self._load_json(session_path)
|
||||
except (ValueError, OSError) as exc:
|
||||
self.logger.warning(f"[{self.name}] skip {session_path.name}: {exc}")
|
||||
continue
|
||||
|
||||
session_id = str(session.get("haystack_session_id") or session_path.stem)
|
||||
session_date = str(session.get("haystack_date") or "").strip()
|
||||
session_dt = self._parse_lme_datetime(session_date)
|
||||
if session_dt is not None and session_dt > question_dt:
|
||||
session_ids_illegal.append(session_id)
|
||||
filtered_sessions.append(
|
||||
{
|
||||
"session_id": session_id,
|
||||
"session_date": session_date,
|
||||
"session_file": session_path.name,
|
||||
"reason": "session_date_after_question_date",
|
||||
},
|
||||
)
|
||||
continue
|
||||
if session_dt is None:
|
||||
self.logger.warning(
|
||||
f"[{self.name}] keep {session_id}: cannot parse haystack_date={session_date!r}",
|
||||
)
|
||||
day = session_dt.strftime("%Y-%m-%d") if session_dt is not None else (self._parse_day(session_date) or "")
|
||||
sessions.append((session, session_path, session_id, session_date, day))
|
||||
|
||||
daily_dir = self.config_value("daily_dir")
|
||||
resource_dir = self._resource_dir_name()
|
||||
start_interval_seconds = float(self.kwargs.get("start_interval_seconds", START_INTERVAL_SECONDS))
|
||||
if start_interval_seconds < 0:
|
||||
start_interval_seconds = START_INTERVAL_SECONDS
|
||||
concurrency = int(self.kwargs.get("concurrency", MAX_CONCURRENCY))
|
||||
if concurrency <= 0:
|
||||
concurrency = MAX_CONCURRENCY
|
||||
concurrency = min(concurrency, MAX_CONCURRENCY)
|
||||
total = len(sessions)
|
||||
self.logger.info(
|
||||
f"[{self.name}] extracting {total} sessions from {session_dir} "
|
||||
f"(filtered {len(session_ids_illegal)} sessions after question_date, "
|
||||
f"start_interval={start_interval_seconds}s, concurrency={concurrency})",
|
||||
def _build_messages(self, raw_messages: list) -> list[Msg]:
|
||||
# Interpolate timestamps: if any message carries created_at, fill in
|
||||
# the rest via linear interpolation so the whole session has coherent
|
||||
# time ordering (see _interpolate_timestamps docstring for rules).
|
||||
interpolated = _interpolate_timestamps(
|
||||
[item if not isinstance(item, dict) else dict(item) for item in raw_messages],
|
||||
)
|
||||
return [self._to_msg(item) for item in interpolated]
|
||||
|
||||
self._reserved.clear()
|
||||
failed_extracts: list[dict] = []
|
||||
retry_initial_seconds = float(self.kwargs.get("retry_initial_seconds", RETRY_INITIAL_SECONDS))
|
||||
retry_max_seconds = float(self.kwargs.get("retry_max_seconds", RETRY_MAX_SECONDS))
|
||||
retry_max_attempts_raw = self.kwargs.get("retry_max_attempts")
|
||||
retry_max_attempts = int(retry_max_attempts_raw) if retry_max_attempts_raw not in (None, "") else 0
|
||||
if retry_initial_seconds <= 0:
|
||||
retry_initial_seconds = RETRY_INITIAL_SECONDS
|
||||
retry_max_seconds = max(retry_max_seconds, retry_initial_seconds)
|
||||
retry_gate = asyncio.Condition()
|
||||
retry_sleeping_extract_idxs: set[int] = set()
|
||||
submit_lock = asyncio.Lock()
|
||||
last_submitted_at = 0.0
|
||||
semaphore = asyncio.Semaphore(concurrency)
|
||||
|
||||
def has_prior_retry_sleeping(idx: int) -> bool:
|
||||
return any(retry_idx < idx for retry_idx in retry_sleeping_extract_idxs)
|
||||
|
||||
async def wait_for_start_slot() -> None:
|
||||
nonlocal last_submitted_at
|
||||
async with submit_lock:
|
||||
sleep_seconds = last_submitted_at + start_interval_seconds - time.monotonic()
|
||||
if sleep_seconds > 0:
|
||||
await asyncio.sleep(sleep_seconds)
|
||||
last_submitted_at = time.monotonic()
|
||||
|
||||
async def wait_for_healthy_start_slot(idx: int, session_id: str) -> None:
|
||||
while True:
|
||||
async with retry_gate:
|
||||
if has_prior_retry_sleeping(idx):
|
||||
self.logger.info(
|
||||
f"[{self.name}] ({idx}/{total}) {session_id} waits for earlier retry sleep",
|
||||
)
|
||||
await retry_gate.wait_for(lambda: not has_prior_retry_sleeping(idx))
|
||||
|
||||
await wait_for_start_slot()
|
||||
|
||||
async with retry_gate:
|
||||
if not has_prior_retry_sleeping(idx):
|
||||
return
|
||||
|
||||
async def mark_retry_sleeping(idx: int) -> None:
|
||||
async with retry_gate:
|
||||
retry_sleeping_extract_idxs.add(idx)
|
||||
retry_gate.notify_all()
|
||||
|
||||
async def mark_retry_awake(idx: int) -> None:
|
||||
async with retry_gate:
|
||||
retry_sleeping_extract_idxs.discard(idx)
|
||||
retry_gate.notify_all()
|
||||
|
||||
async def reply_with_retry(idx: int, user_prompt: str, session_id: str) -> dict:
|
||||
attempt = 1
|
||||
sleep_seconds = retry_initial_seconds
|
||||
while True:
|
||||
try:
|
||||
await wait_for_healthy_start_slot(idx, session_id)
|
||||
result = await self.agent_wrapper.reply(
|
||||
user_prompt,
|
||||
system_prompt=self.get_prompt("system_prompt"),
|
||||
output_schema=_MEMORY_SCHEMA,
|
||||
)
|
||||
if not isinstance(result.get("structured_output"), dict):
|
||||
raise ValueError("agent reply missing structured_output")
|
||||
await mark_retry_awake(idx)
|
||||
if attempt > 1:
|
||||
self.logger.info(f"[{self.name}] extract recovered for {session_id} after {attempt} attempts")
|
||||
return result
|
||||
except Exception as exc:
|
||||
if self._is_data_inspection_error(exc):
|
||||
await mark_retry_awake(idx)
|
||||
raise
|
||||
if 0 < retry_max_attempts <= attempt:
|
||||
await mark_retry_awake(idx)
|
||||
raise
|
||||
await mark_retry_sleeping(idx)
|
||||
next_sleep = min(sleep_seconds, retry_max_seconds)
|
||||
self.logger.warning(
|
||||
f"[{self.name}] extract attempt {attempt} failed for {session_id}: {exc}; "
|
||||
f"retrying in {next_sleep:.1f}s",
|
||||
)
|
||||
await asyncio.sleep(next_sleep)
|
||||
await mark_retry_awake(idx)
|
||||
sleep_seconds = min(sleep_seconds * 2, retry_max_seconds)
|
||||
attempt += 1
|
||||
|
||||
async def extract_one(
|
||||
idx: int,
|
||||
session: dict,
|
||||
session_path: Path,
|
||||
session_id: str,
|
||||
session_date: str,
|
||||
day: str,
|
||||
) -> dict | None:
|
||||
if not day:
|
||||
self.logger.warning(f"[{self.name}] skip {session_id}: unparseable date {session_date!r}")
|
||||
return None
|
||||
messages = session.get("messages") or []
|
||||
|
||||
user_prompt = self.prompt_format(
|
||||
"user_message",
|
||||
session_id=session_id,
|
||||
session_date=session_date,
|
||||
messages=self._format_messages(messages),
|
||||
)
|
||||
try:
|
||||
result = await reply_with_retry(idx, user_prompt, session_id)
|
||||
except Exception as exc: # noqa: BLE001 — one bad session must not abort the sweep
|
||||
if self._is_data_inspection_error(exc):
|
||||
self.logger.warning(
|
||||
f"[{self.name}] extract fallback for {session_id}: non-retryable data inspection error",
|
||||
)
|
||||
failed_extracts.append(
|
||||
{
|
||||
"session_id": session_id,
|
||||
"session_date": session_date,
|
||||
"session_file": session_path.name,
|
||||
"error": str(exc),
|
||||
"non_retryable": True,
|
||||
"fallback": True,
|
||||
"fallback_reason": "data_inspection_failed",
|
||||
"raw_session": session,
|
||||
},
|
||||
)
|
||||
return None
|
||||
self.logger.warning(f"[{self.name}] extract failed for {session_id}: {exc}")
|
||||
failed_extracts.append(
|
||||
{
|
||||
"session_id": session_id,
|
||||
"session_date": session_date,
|
||||
"session_file": session_path.name,
|
||||
"error": str(exc),
|
||||
"non_retryable": False,
|
||||
"fallback": False,
|
||||
},
|
||||
)
|
||||
return None
|
||||
|
||||
extracted = result.get("structured_output")
|
||||
name = description = body = ""
|
||||
if isinstance(extracted, dict):
|
||||
name = str(extracted.get("name") or "").strip()
|
||||
description = str(extracted.get("description") or "").strip()
|
||||
body = str(extracted.get("body") or "").strip()
|
||||
if not isinstance(extracted, dict) or not name or not body:
|
||||
if isinstance(extracted, dict):
|
||||
self.logger.info(f"[{self.name}] empty extraction for {session_id}; skipping")
|
||||
else:
|
||||
self.logger.warning(f"[{self.name}] no structured output for {session_id}; skipping")
|
||||
return None
|
||||
|
||||
unique_name = await self._reserve_name(daily_dir, day, name, session_id)
|
||||
rel_path = f"{daily_dir}/{day}/{unique_name}.md"
|
||||
post = frontmatter.Post(
|
||||
body,
|
||||
name=unique_name,
|
||||
description=description,
|
||||
session_id=session_id,
|
||||
session_date=session_date,
|
||||
source=f"[[{resource_dir}/{session_path.name}]]",
|
||||
)
|
||||
abs_path = self.workspace_path / rel_path
|
||||
abs_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
abs_path.write_text(frontmatter.dumps(post), encoding="utf-8")
|
||||
|
||||
self.logger.info(f"[{self.name}] ({idx}/{total}) {session_id} -> {rel_path}")
|
||||
return {"session_id": session_id, "date": day, "path": rel_path}
|
||||
|
||||
async def extract_one_limited(
|
||||
idx: int,
|
||||
session: dict,
|
||||
session_path: Path,
|
||||
session_id: str,
|
||||
session_date: str,
|
||||
day: str,
|
||||
) -> dict | None:
|
||||
async with semaphore:
|
||||
return await extract_one(idx, session, session_path, session_id, session_date, day)
|
||||
|
||||
results = await asyncio.gather(
|
||||
*(
|
||||
extract_one_limited(idx, session, session_path, session_id, session_date, day)
|
||||
for idx, (session, session_path, session_id, session_date, day) in enumerate(sessions, start=1)
|
||||
),
|
||||
)
|
||||
written = [r for r in results if r is not None]
|
||||
fallback_extracts = [e for e in failed_extracts if e.get("fallback")]
|
||||
|
||||
self.context.response.success = True
|
||||
self.context.response.answer = f"wrote {len(written)}/{total} session notes"
|
||||
self.context.response.metadata.update(
|
||||
{
|
||||
"num_sessions": total,
|
||||
"num_session_files": len(session_files),
|
||||
"num_written": len(written),
|
||||
"num_failed_extracts": len(failed_extracts),
|
||||
"num_fallback_extracts": len(fallback_extracts),
|
||||
"num_filtered_sessions": len(session_ids_illegal),
|
||||
"session_ids_illegal": session_ids_illegal,
|
||||
"filtered_sessions": filtered_sessions,
|
||||
"failed_extracts": failed_extracts,
|
||||
"fallback_extracts": fallback_extracts,
|
||||
"notes": written,
|
||||
},
|
||||
)
|
||||
return self.context.response
|
||||
def _reply_extra_kwargs(self, day: str) -> dict:
|
||||
return {"tool_defaults": {"daily_write": {"date": day}}}
|
||||
|
|
|
|||
|
|
@ -1,45 +1,334 @@
|
|||
system_prompt: |
|
||||
You are a memory extractor for the LongMemEval benchmark. You are given ONE chat session
|
||||
between a user and an assistant. Your job is to extract its content COMPLETELY into a single
|
||||
daily memory note that will later be retrieved by both BM25 keyword search and dense vector
|
||||
search. Completeness and retrievability are the only goals — do NOT compress or editorialize.
|
||||
You are an automatic memory extraction system. Your task is to extract ALL valuable information from conversations and record them into a structured daily memory note. Ensure **complete coverage** — every piece of information worth preserving must be captured.
|
||||
|
||||
## What to extract — everything that could ever be asked later
|
||||
Capture every core fact stated or clearly implied in the session, including but not limited to:
|
||||
- Facts about the user: identity, preferences, habits, possessions, relationships, plans, goals.
|
||||
- Events and actions: what happened, what was decided, what the user did or intends to do.
|
||||
- Entities: people, places, organizations, products, titles, brands — with their exact names.
|
||||
- Numbers and quantities: durations, distances, prices, counts, measurements — verbatim.
|
||||
- Time information ABOVE ALL: absolute dates, weekdays, and relative expressions ("last week",
|
||||
"since January 15th", "every morning", "for 3 years"). Always keep the fact together with its
|
||||
time expression, and when possible also anchor it to the session date.
|
||||
## Extraction Principles
|
||||
|
||||
Do not invent anything. Only record what is actually in the session. If the session is pure
|
||||
small talk with no durable facts, still produce a minimal faithful note (do not fabricate).
|
||||
1. **Complete Coverage**: Generate enough content to ensure ALL information in the dialogues is captured. Missing information is worse than redundancy.
|
||||
2. **Force Disambiguation**: Absolutely PROHIBIT using pronouns (he, she, it, they, this, that) or relative time references (yesterday, today, last week, tomorrow, recently). Always use full names, explicit dates, and specific references.
|
||||
3. **Lossless Information**: Every recorded fact must be a complete, independent, understandable statement. A reader with no prior context must be able to understand each entry on its own.
|
||||
4. **Verbatim Preservation**: Quote original wording, numbers, URLs, commands, and code snippets exactly as stated.
|
||||
|
||||
## How to write the body — optimize for search
|
||||
- Write plain natural-language declarative sentences (one fact per sentence or bullet). Dense
|
||||
retrievers embed sentences well; BM25 matches exact tokens — so both benefit from full,
|
||||
unabbreviated wording.
|
||||
- State entities, dates and numbers explicitly and verbatim; expand abbreviations and also
|
||||
include common aliases/synonyms the user or assistant used, so keyword search can hit them.
|
||||
- Prefer the user's own key phrasing at important points (quote short fragments verbatim).
|
||||
- Use markdown structure (headings/bullets) freely, but never drop a fact for the sake of brevity.
|
||||
## What to Extract
|
||||
|
||||
## Frontmatter fields you return
|
||||
- `name`: a concise, stable, kebab-case topic/event stem (no date, no slashes or reserved
|
||||
characters). It is only a filename — the searchable content lives in `description` and `body`.
|
||||
- `description`: a thorough, search-friendly abstract that on its own conveys all key facts.
|
||||
- `body`: the complete extraction as described above.
|
||||
- **Persons**: All person names mentioned (full names when available)
|
||||
- **Entities**: Companies, products, organizations, projects, repositories
|
||||
- **Decisions & Events**: What happened, what was decided, and why
|
||||
- **Facts & Preferences**: Persistent facts about the user — identity, habits, preferences, constraints
|
||||
- **Procedures & Solutions**: Actionable steps, commands, configurations that can be directly reused
|
||||
- **Current State**: Progress, blockers, next steps, open questions
|
||||
- **Temporal Information**: Use absolute dates in ISO 8601 format (YYYY-MM-DD) when explicit time is mentioned
|
||||
- **Locations**: Specific location names when mentioned
|
||||
- **Topic Keywords**: Core keywords that identify the subject matter
|
||||
|
||||
user_message: |
|
||||
Session id: {session_id}
|
||||
Session date: {session_date}
|
||||
## Body Format
|
||||
|
||||
--- Session messages ---
|
||||
{messages}
|
||||
--- End of session ---
|
||||
Use structured sections with headings. Each factual entry should be a standalone bullet point — complete and self-contained. Group entries by topic. Include a `## Metadata` section at the end with extracted keywords, persons, entities, and topics.
|
||||
|
||||
Extract this session COMPLETELY into one search-friendly daily note. Keep every entity, number,
|
||||
and (above all) every time expression verbatim, each attached to the fact it belongs to. Return
|
||||
`name`, `description`, and `body`.
|
||||
## Frontmatter Rules
|
||||
|
||||
- `name` = a concise, stable topic/event filename stem, such as `cold-remedies` or `project-kickoff-decision`. Do not include today's date or the daily directory date; the outer daily path already records the date. For existing notes, update it when a better filename is clearly warranted.
|
||||
- `description` = a thorough, lossless summary; vague descriptions like "notes" / "misc" are unacceptable. The description alone should convey all key information.
|
||||
- **Never set `status`** — it is a field reserved for downstream processing.
|
||||
|
||||
## Tool Call Format — CRITICAL
|
||||
|
||||
You have access to tools (e.g. `daily_write`, `read`, `edit`, `write`, `frontmatter_update`). You MUST invoke them as **structured tool calls** (function calls), NOT as plain text output.
|
||||
|
||||
- **NEVER** write tool invocations as text strings such as `daily_write name=...`. Text output does NOT execute tools — no memory will be saved.
|
||||
- **ALWAYS** use the tool call mechanism provided by the runtime. The system handles parameter passing automatically.
|
||||
- If you write a tool name and its parameters as text, NO tool will be executed and NO memory will be saved.
|
||||
system_prompt_zh: |
|
||||
你是自动记忆提取系统。你的任务是从对话中提取所有有价值的信息，并记录到结构化的日记记忆中。确保**完整覆盖**——每一条值得保留的信息都必须被捕获。
|
||||
|
||||
## 提取原则
|
||||
|
||||
1. **完整覆盖**：生成足够的内容以确保对话中的所有信息都被捕获。遗漏信息比冗余更糟。
|
||||
2. **强制消歧**：绝对禁止使用代词（他、她、它、他们、这个、那个）或相对时间（昨天、今天、上周、明天、最近）。必须使用全名、明确日期和具体引用。
|
||||
3. **无损信息**：每条记录的事实必须是完整的、独立的、可理解的陈述。没有上下文的读者必须能单独理解每条记录。
|
||||
4. **原文保留**：逐字引用原始措辞、数字、URL、命令和代码片段。
|
||||
|
||||
## 提取什么
|
||||
|
||||
- **人物**：提及的所有人名（尽量使用全名）
|
||||
- **实体**：公司、产品、组织、项目、仓库
|
||||
- **决策与事件**：发生了什么、决定了什么、为什么
|
||||
- **事实与偏好**：关于用户的持久事实——身份、习惯、偏好、约束
|
||||
- **流程与方案**：可直接复用的操作步骤、命令、配置
|
||||
- **当前状态**：进度、卡点、下一步、未决问题
|
||||
- **时间信息**：当对话中明确提及时间时，使用 ISO 8601 绝对日期格式（YYYY-MM-DD）
|
||||
- **地点**：提及的具体地点名称
|
||||
- **主题关键词**：标识主题的核心关键词
|
||||
|
||||
## 正文格式
|
||||
|
||||
使用带标题的结构化分节。每条事实记录应是独立的列表项——完整且自包含。按主题分组。在末尾添加 `## Metadata` 部分，列出提取的关键词、人物、实体和主题。
|
||||
|
||||
## Frontmatter 规则
|
||||
|
||||
- `name` = 简洁、稳定的主题/事件文件名 stem，例如 `cold-remedies` 或 `project-kickoff-decision`。不要包含今天日期或日记目录日期；外层日记路径已经记录日期。对已有笔记，如果明显有更好的文件名，就更新它。
|
||||
- `description` = 详尽的无损总结；模糊的描述如 "notes" / "misc" 不可接受。仅凭 description 就应能传达全部核心信息。
|
||||
- **永远不要设置 `status`**——它是下游处理保留的字段。
|
||||
|
||||
## 工具调用格式 — 关键
|
||||
|
||||
你可以使用工具（如 `daily_write`、`read`、`edit`、`write`、`frontmatter_update`）。你**必须**以**结构化工具调用**（函数调用）的方式调用它们，而不是以纯文本输出。
|
||||
|
||||
- **绝不**将工具调用作为文本字符串输出，如 `daily_write name=...`。文本输出**不会**执行工具——不会保存任何记忆。
|
||||
- **始终**使用运行时提供的工具调用机制。系统会自动处理参数传递。
|
||||
- 如果你将工具名和参数作为文本输出，**不会有任何工具被执行**，也**不会**保存任何记忆。
|
||||
|
||||
user_message_create: |
|
||||
Today: {today}
|
||||
Extra hint: {note}
|
||||
Session ID: {session_id}
|
||||
|
||||
# Recent Conversation
|
||||
|
||||
{history}
|
||||
|
||||
# Your Task
|
||||
|
||||
Extract all valuable information from the conversation above and record them into a new daily memory note.
|
||||
|
||||
## Step 1 — Skip Check
|
||||
|
||||
Did the conversation produce substantive information worth long-term memory? Pure greetings or small talk → reply with a brief skip message and stop (do not call any tools).
|
||||
|
||||
When truly ambiguous, default to writing — losing a memory is worse than writing one extra note.
|
||||
|
||||
## Step 2 — Extract & Write
|
||||
|
||||
Analyze the conversation and extract:
|
||||
- **keywords**: Core keywords (names, places, entities, topic words)
|
||||
- **persons**: All person names mentioned
|
||||
- **entities**: Companies, products, organizations, projects
|
||||
- **topic**: The main topic(s) of the conversation
|
||||
- **timestamp**: Absolute time in ISO 8601 format (if explicit time mentioned)
|
||||
- **location**: Specific location name (if mentioned)
|
||||
|
||||
Then create the note by **calling the `daily_write` tool** with these parameters:
|
||||
- `name`: <a concise, stable topic/event filename stem in kebab-case>
|
||||
- `description`: <a thorough, lossless summary>
|
||||
- `session_id`: {session_id}
|
||||
- `date`: {today}
|
||||
- `content`: <the full body text>
|
||||
|
||||
⚠️ CRITICAL: You MUST invoke `daily_write` as a **tool call** (function call). Do NOT write `daily_write name=...` as plain text — that will NOT execute the tool and NO memory will be saved.
|
||||
|
||||
Body requirements:
|
||||
- Each factual entry must be a complete, independent, self-contained statement (lossless restatement).
|
||||
- PROHIBIT pronouns (he/she/it/they/this/that) — use full names and explicit references.
|
||||
- PROHIBIT relative time (yesterday/today/last week) — use absolute dates (YYYY-MM-DD).
|
||||
- Group entries by topic with headings.
|
||||
- End with a `## Metadata` section containing: keywords, persons, entities, topic.
|
||||
|
||||
Filename and description rules:
|
||||
- `name`: concise, stable topic/event filename stem in kebab-case. No date, no slash/backslash, no `< > : " | ? *`.
|
||||
- `description`: a thorough, lossless summary — specific enough that the description alone conveys all key information.
|
||||
|
||||
## Step 3 — Summary
|
||||
|
||||
State in one sentence what you did (which file was created). This is your final text output.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Create at most 3 notes for this session. If the conversation covers multiple distinct topics, split them into separate notes (one topic per note). If only one topic is discussed, create just one note.
|
||||
user_message_create_zh: |
|
||||
今天：{today}
|
||||
额外提示：{note}
|
||||
Session ID：{session_id}
|
||||
|
||||
# 最近的对话
|
||||
|
||||
{history}
|
||||
|
||||
# 你的任务
|
||||
|
||||
从上述对话中提取所有有价值的信息，并记录到一条新的日记记忆中。
|
||||
|
||||
## 步骤 1 — 跳过检查
|
||||
|
||||
对话是否产生了值得长期记忆的实质性信息？纯粹的寒暄或闲聊 → 回复一条简短的跳过消息并停止（不调用任何工具）。
|
||||
|
||||
当真正模棱两可时，默认写入——丢失记忆比多写一条笔记更糟。
|
||||
|
||||
## 步骤 2 — 提取并写入
|
||||
|
||||
分析对话并提取：
|
||||
- **keywords**：核心关键词（人名、地点、实体、主题词）
|
||||
- **persons**：提及的所有人名
|
||||
- **entities**：公司、产品、组织、项目
|
||||
- **topic**：对话的主要主题
|
||||
- **timestamp**：ISO 8601 格式的绝对时间（如果对话中明确提及时间）
|
||||
- **location**：具体地点名称（如果提及）
|
||||
|
||||
然后**调用 `daily_write` 工具**，使用以下参数一次性创建笔记：
|
||||
- `name`：<简洁、稳定的主题/事件文件名 stem，采用 kebab-case>
|
||||
- `description`：<详尽的无损总结>
|
||||
- `session_id`：{session_id}
|
||||
- `date`：{today}
|
||||
- `content`：<完整正文>
|
||||
|
||||
⚠️ 关键：你**必须**以**工具调用**（函数调用）方式调用 `daily_write`。不要将 `daily_write name=...` 作为纯文本输出——那样**不会**执行工具，也**不会**保存任何记忆。
|
||||
|
||||
正文要求：
|
||||
- 每条事实记录必须是完整的、独立的、自包含的陈述（无损复述）。
|
||||
- 禁止使用代词（他/她/它/他们/这个/那个）——必须使用全名和明确引用。
|
||||
- 禁止使用相对时间（昨天/今天/上周）——必须使用绝对日期（YYYY-MM-DD）。
|
||||
- 按主题分组并使用标题。
|
||||
- 末尾添加 `## Metadata` 部分，包含：keywords、persons、entities、topic。
|
||||
|
||||
文件名和描述规则：
|
||||
- `name`：简洁、稳定的主题/事件文件名 stem，采用 kebab-case。不含日期、不含 slash/反斜杠、不含 `< > : " | ? *`。
|
||||
- `description`：详尽的无损总结——具体到仅凭 description 就能传达全部核心信息。
|
||||
|
||||
## 步骤 3 — 总结
|
||||
|
||||
用一句话说明你做了什么（创建了哪个文件）。这是你最后一次文本输出。
|
||||
|
||||
## 边界
|
||||
|
||||
- 当前 session 最多创建 3 条笔记。如果对话涵盖多个不同主题，将其拆分为独立笔记（每个主题一条）。如果只讨论了一个主题，则只创建一条笔记。
|
||||
|
||||
user_message_update: |
|
||||
Today: {today}
|
||||
Extra hint: {note}
|
||||
Target path: {note_path}
|
||||
|
||||
# Recent Conversation
|
||||
|
||||
{history}
|
||||
|
||||
# Your Task
|
||||
|
||||
Extract all valuable information from the conversation above and merge them into the existing daily note at the target path.
|
||||
|
||||
## Step 1 — Skip Check
|
||||
|
||||
Did the conversation produce substantive information worth long-term memory? Pure greetings or small talk → reply with a brief skip message and stop (do not call any tools).
|
||||
|
||||
When truly ambiguous, default to writing — losing a memory is worse than writing one extra note.
|
||||
|
||||
## Step 2 — Read Existing Content
|
||||
|
||||
**Call the `read` tool** with parameter `path` set to `{note_path}` to inspect the current note content. You MUST invoke it as a tool call, not as text.
|
||||
- If the body is empty (only frontmatter, no actual content) → treat as new, jump to **Step 3b**.
|
||||
- If there is body content → go to **Step 3a** to merge.
|
||||
|
||||
## Step 3a — Merge Update
|
||||
|
||||
The note already has content. Extract new information from the conversation and merge into it.
|
||||
|
||||
Extraction rules (apply to ALL new entries):
|
||||
- Each new fact must be a complete, independent, self-contained statement (lossless restatement).
|
||||
- PROHIBIT pronouns (he/she/it/they/this/that) — use full names and explicit references.
|
||||
- PROHIBIT relative time (yesterday/today/last week) — use absolute dates (YYYY-MM-DD).
|
||||
|
||||
Merge rules:
|
||||
- **Timeline / history entries**: append only, never delete existing entries.
|
||||
- **Current-state entries** (progress, blockers, next steps, open questions): rewrite the entire section to reflect the latest snapshot.
|
||||
- **Everything else**: merge and deduplicate — keep all old facts, add new facts, remove exact duplicates.
|
||||
- **Metadata section**: merge keywords, persons, entities, topics — union of old and new values.
|
||||
|
||||
Execution:
|
||||
1. **Call the `edit` tool** for each section that needs updating, with parameters: `path`="{note_path}", `old`=<original fragment>, `new`=<replacement fragment>. You may call `edit` multiple times. Each call MUST be a tool call, not text.
|
||||
2. After body changes, **call the `frontmatter_update` tool** to refresh frontmatter, with parameters: `path`="{note_path}", `metadata`={{"name": "<updated filename stem>", "description": "<updated summary>"}}.
|
||||
- Keep the existing `name` only when it is already the best concise topic/event filename stem. The system will rename the file after your final response.
|
||||
- Do not add today's date or the daily directory date to `name`.
|
||||
3. If `edit` fails repeatedly (e.g., cannot find the original text due to formatting mismatch), fall back to **calling the `write` tool** with parameters: `path`="{note_path}", `name`=<name>, `description`=<description>, `content`=<full body> for a complete rewrite.
|
||||
|
||||
## Step 3b — Full Write (Empty File Fallback)
|
||||
|
||||
The file exists but its body is empty. Extract and write the full content by **calling the `write` tool** with parameters: `path`="{note_path}", `name`=<name>, `description`=<description>, `content`=<body>.
|
||||
|
||||
Body requirements:
|
||||
- Each factual entry must be a complete, independent, self-contained statement.
|
||||
- PROHIBIT pronouns — use full names and explicit references.
|
||||
- PROHIBIT relative time — use absolute dates (YYYY-MM-DD).
|
||||
- Group entries by topic with headings.
|
||||
- End with a `## Metadata` section containing: keywords, persons, entities, topic.
|
||||
|
||||
Filename and description rules:
|
||||
- `name`: concise, stable topic/event filename stem in kebab-case. No date.
|
||||
- `description`: a thorough, lossless summary — the description alone should convey all key information.
|
||||
|
||||
## Step 4 — Summary
|
||||
|
||||
State in one sentence what you did (what content was updated). This is your final text output.
|
||||
|
||||
## Boundaries
|
||||
|
||||
- Only operate on one target path: `{note_path}`. Do not touch other notes.
|
||||
- `write` unconditionally overwrites body and frontmatter — use with caution.
|
||||
- Filename changes are expressed by updating frontmatter `name`.
|
||||
user_message_update_zh: |
|
||||
今天：{today}
|
||||
额外提示：{note}
|
||||
目标路径：{note_path}
|
||||
|
||||
# 最近的对话
|
||||
|
||||
{history}
|
||||
|
||||
# 你的任务
|
||||
|
||||
从上述对话中提取所有有价值的信息，并合并到目标路径的已有日记中。
|
||||
|
||||
## 步骤 1 — 跳过检查
|
||||
|
||||
对话是否产生了值得长期记忆的实质性信息？纯粹的寒暄或闲聊 → 回复一条简短的跳过消息并停止（不调用任何工具）。
|
||||
|
||||
当真正模棱两可时，默认写入——丢失记忆比多写一条笔记更糟。
|
||||
|
||||
## 步骤 2 — 读取现有内容
|
||||
|
||||
**调用 `read` 工具**，参数 `path` 设为 `{note_path}`，查看当前笔记内容。必须以工具调用方式调用，不要作为文本输出。
|
||||
- 如果正文为空（只有 frontmatter 无实际内容）→ 按新建处理，跳到 **步骤 3b**。
|
||||
- 如果有正文内容 → 转到 **步骤 3a** 进行合并。
|
||||
|
||||
## 步骤 3a — 合并更新
|
||||
|
||||
笔记已有内容。从对话中提取新信息并合并进去。
|
||||
|
||||
提取规则（适用于所有新条目）：
|
||||
- 每条新事实必须是完整的、独立的、自包含的陈述（无损复述）。
|
||||
- 禁止使用代词（他/她/它/他们/这个/那个）——必须使用全名和明确引用。
|
||||
- 禁止使用相对时间（昨天/今天/上周）——必须使用绝对日期（YYYY-MM-DD）。
|
||||
|
||||
合并规则：
|
||||
- **时间线 / 历史条目**：仅追加，永远不删除已有条目。
|
||||
- **当下状态类条目**（进度、卡点、下一步、未决问题）：整段重写，反映最新快照。
|
||||
- **其余内容**：合并去重——保留全部旧事实，添加新事实，去除完全重复项。
|
||||
- **Metadata 部分**：合并 keywords、persons、entities、topics——取旧值和新值的并集。
|
||||
|
||||
执行：
|
||||
1. **调用 `edit` 工具**更新需要修改的部分，参数为：`path`="{note_path}"、`old`=<原文片段>、`new`=<替换片段>。可以多次调用 `edit`。每次调用必须是工具调用，不是文本。
|
||||
2. 正文变更后，**调用 `frontmatter_update` 工具**刷新 frontmatter，参数为：`path`="{note_path}"、`metadata`={{"name": "<更新后的文件名 stem>", "description": "<更新后的总结>"}}。
|
||||
- 只有当前 `name` 已经是最合适的简洁主题/事件文件名 stem 时才保留。系统会在你最终回复后负责重命名文件。
|
||||
- 不要在 `name` 中加入今天日期或日记目录日期。
|
||||
3. 如果 `edit` 多次失败（如因格式不匹配找不到原文），退回**调用 `write` 工具**全量重写，参数为：`path`="{note_path}"、`name`=<name>、`description`=<description>、`content`=<完整正文>。
|
||||
|
||||
## 步骤 3b — 全量写入（空文件 fallback）
|
||||
|
||||
文件存在但正文为空。提取并**调用 `write` 工具**一次性写入完整内容，参数为：`path`="{note_path}"、`name`=<name>、`description`=<description>、`content`=<正文>。
|
||||
|
||||
正文要求：
|
||||
- 每条事实记录必须是完整的、独立的、自包含的陈述。
|
||||
- 禁止使用代词——必须使用全名和明确引用。
|
||||
- 禁止使用相对时间——必须使用绝对日期（YYYY-MM-DD）。
|
||||
- 按主题分组并使用标题。
|
||||
- 末尾添加 `## Metadata` 部分，包含：keywords、persons、entities、topic。
|
||||
|
||||
文件名和描述规则：
|
||||
- `name`：简洁、稳定的主题/事件文件名 stem，采用 kebab-case。不含日期。
|
||||
- `description`：详尽的无损总结——仅凭 description 就应能传达全部核心信息。
|
||||
|
||||
## 步骤 4 — 总结
|
||||
|
||||
用一句话说明你做了什么（更新了哪些内容）。这是你最后一次文本输出。
|
||||
|
||||
## 边界
|
||||
|
||||
- 只针对一个目标路径：`{note_path}`。不要碰其他笔记。
|
||||
- `write` 会无条件覆盖正文和 frontmatter，请谨慎使用。
|
||||
- 文件名变化通过更新 frontmatter `name` 表达。
|
||||
|
|
|
|||
|
|
@ -1,45 +0,0 @@
|
|||
"""Answer a query directly from the supplied session context."""
|
||||
|
||||
from ...base_step import BaseStep
|
||||
from ....components import R
|
||||
|
||||
|
||||
@R.register("context_answer_step")
|
||||
class ContextAnswerStep(BaseStep):
|
||||
"""Answer a query using the LongMemEval direct-reading prompt."""
|
||||
|
||||
async def execute(self):
|
||||
assert self.context is not None
|
||||
query: str = self.context.get("query", "")
|
||||
session_context: str = self.context.get("session_context", "")
|
||||
current_date: str = self.context.get("current_date", "")
|
||||
|
||||
if not query:
|
||||
raise ValueError("context_answer_step requires non-empty query")
|
||||
if not session_context:
|
||||
raise ValueError("context_answer_step requires non-empty session_context")
|
||||
if self.agent_wrapper is None:
|
||||
raise ValueError("context_answer_step requires agent_wrapper")
|
||||
|
||||
user_prompt = self.prompt_format(
|
||||
"user_message",
|
||||
session_context=session_context,
|
||||
current_date=current_date,
|
||||
query=query,
|
||||
)
|
||||
result = await self.agent_wrapper.reply(user_prompt)
|
||||
answer = (result.get("result") or "").strip()
|
||||
|
||||
self.logger.info(f"[{self.name}] context answer: {answer}")
|
||||
self.context["context_answer"] = answer
|
||||
self.context.response.success = True
|
||||
self.context.response.answer = answer
|
||||
self.context.response.metadata.update(
|
||||
{
|
||||
"query": query,
|
||||
"session_context": session_context,
|
||||
"current_date": current_date,
|
||||
"context_answer": answer,
|
||||
},
|
||||
)
|
||||
return self.context.response
|
||||
|
|
@ -1,9 +0,0 @@
|
|||
user_message: |
|
||||
I will give you several history chats between you and a user. Please answer the question
|
||||
based on the relevant chat history. Answer the question step by step: first extract all the
|
||||
relevant information, and then reason over the information to get the answer.
|
||||
|
||||
History Chats: {session_context}
|
||||
Current Date: {current_date}
|
||||
Question: {query}
|
||||
Answer (step by step):
|
||||
|
|
@ -1,103 +0,0 @@
|
|||
"""extract_session_by_id — deep-read one raw session, keyed by its session_id.
|
||||
|
||||
This is the hand-written function tool the answering agent sees. Search results
|
||||
surface a note's ``session_id``; when a hit looks relevant, the agent passes that
|
||||
``session_id`` here. The step resolves the question/time from ``query.json``,
|
||||
locates the raw session file (named ``<date>_(...)_<time>@<session_id>.json``
|
||||
under ``resource_dir``), loads its messages, and asks an agent to extract —
|
||||
completely and verbatim — every part of that session relevant to the question.
|
||||
"""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from ...base_step import BaseStep
|
||||
from ....components import R
|
||||
|
||||
|
||||
@R.register("lme_extract_session_step")
|
||||
class LmeExtractSessionStep(BaseStep):
|
||||
"""Resolve a session_id to raw content, then deep-read it for the question."""
|
||||
|
||||
def _resource_dir_name(self) -> str:
|
||||
return self.app_context.app_config.resource_dir if self.app_context is not None else "session"
|
||||
|
||||
def _load_query(self) -> dict:
|
||||
path = self.workspace_path / "query.json"
|
||||
with path.open(encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError("query.json is not a JSON object")
|
||||
return data
|
||||
|
||||
def _find_session_file(self, session_id: str) -> Path | None:
|
||||
session_dir = self.workspace_path / self._resource_dir_name()
|
||||
if not session_dir.is_dir():
|
||||
return None
|
||||
# Files are named "<date>_(...)_<time>@<session_id>.json".
|
||||
matches = list(session_dir.glob(f"*@{session_id}.json"))
|
||||
if matches:
|
||||
return matches[0]
|
||||
# Fall back to a plain "<session_id>.json" naming.
|
||||
direct = session_dir / f"{session_id}.json"
|
||||
return direct if direct.is_file() else None
|
||||
|
||||
async def execute(self):
|
||||
assert self.context is not None
|
||||
if self.agent_wrapper is None:
|
||||
raise ValueError("lme_extract_session_step requires agent_wrapper")
|
||||
|
||||
session_id: str = str(self.context.get("session_id", "") or "").strip()
|
||||
if not session_id:
|
||||
self.context.response.success = False
|
||||
self.context.response.answer = "Error: session_id is required"
|
||||
return self.context.response
|
||||
|
||||
try:
|
||||
query = self._load_query()
|
||||
except (OSError, ValueError) as exc:
|
||||
self.context.response.success = False
|
||||
self.context.response.answer = f"Error: cannot read query.json: {exc}"
|
||||
return self.context.response
|
||||
question = str(query.get("question", "") or "").strip()
|
||||
question_time = str(query.get("question_date", "") or "").strip()
|
||||
|
||||
session_path = self._find_session_file(session_id)
|
||||
if session_path is None:
|
||||
self.context.response.success = False
|
||||
self.context.response.answer = (
|
||||
f"Error: no session file found for session_id={session_id!r}. "
|
||||
"Use a session_id shown in a search result."
|
||||
)
|
||||
return self.context.response
|
||||
|
||||
try:
|
||||
with session_path.open(encoding="utf-8") as f:
|
||||
session = json.load(f)
|
||||
except (OSError, ValueError) as exc:
|
||||
self.context.response.success = False
|
||||
self.context.response.answer = f"Error: cannot read session {session_path.name}: {exc}"
|
||||
return self.context.response
|
||||
|
||||
messages = session.get("messages") if isinstance(session, dict) else None
|
||||
session_content = json.dumps(messages or session, ensure_ascii=False, indent=2)
|
||||
|
||||
user_prompt = self.prompt_format(
|
||||
"user_message",
|
||||
question=question or "(unknown)",
|
||||
question_time=question_time or "(unknown)",
|
||||
session_content=session_content,
|
||||
)
|
||||
result = await self.agent_wrapper.reply(
|
||||
user_prompt,
|
||||
system_prompt=self.get_prompt("system_prompt"),
|
||||
)
|
||||
answer = (result.get("result") or "").strip()
|
||||
|
||||
self.logger.info(f"[{self.name}] extracted {len(answer)} chars for session_id={session_id!r}")
|
||||
self.context.response.success = True
|
||||
self.context.response.answer = answer
|
||||
self.context.response.metadata.update(
|
||||
{"session_id": session_id, "session_file": session_path.name},
|
||||
)
|
||||
return self.context.response
|
||||
|
|
@ -1,27 +0,0 @@
|
|||
system_prompt: |
|
||||
You are a precise evidence extractor for the LongMemEval benchmark. You are given ONE full
|
||||
chat session (raw messages) together with a target question and the time the question was asked.
|
||||
Your job is to extract, COMPLETELY and VERBATIM, every part of this session that is relevant to
|
||||
answering the question.
|
||||
|
||||
Rules:
|
||||
- Extract the actual content from the session — quote the relevant user/assistant statements as
|
||||
they appear. Do not summarize away details, and do not invent anything not in the session.
|
||||
- Preserve all time information exactly: absolute dates, weekdays, and relative expressions
|
||||
("last week", "since January 15th", "every day", durations, frequencies). Keep each fact
|
||||
together with its time expression, and relate it to the question time when that matters
|
||||
(e.g. a fact stated before the question date is valid evidence; note any date conflicts).
|
||||
- Keep entities, names, and numbers verbatim.
|
||||
- If, after reading the whole session, nothing in it is relevant to the question, reply with
|
||||
exactly: NOT RELEVANT
|
||||
|
||||
user_message: |
|
||||
Question: {question}
|
||||
Question asked at: {question_time}
|
||||
|
||||
--- Full session content ---
|
||||
{session_content}
|
||||
--- End of session ---
|
||||
|
||||
Extract everything in this session that is relevant to answering the question, verbatim and with
|
||||
all time information preserved. If nothing is relevant, reply exactly: NOT RELEVANT
|
||||
|
|
@ -1,280 +0,0 @@
|
|||
"""Produce a final, evidence-backed answer for a LongMemEval case.
|
||||
|
||||
The step puts the complete query, golden-answer object, and any available
|
||||
disputed reference answers directly into the prompt. Raw session content stays out of the model
|
||||
context: Claude Code starts in the sample's ``session`` directory and uses its
|
||||
normal file tools to inspect whichever sessions it needs. Session timestamps
|
||||
are scanned only to identify evidence that did not exist at question time;
|
||||
``answer_session_ids`` are not evaluated.
|
||||
|
||||
Claude Code is intentionally used without an output schema. Its ordinary text
|
||||
reply may contain narration but must include exactly one fenced ``json`` block
|
||||
whose object contains ``reason``, ``golden_answer_correct``, ``answer``, and
|
||||
``is_session_time_wrong``. API errors and invalid replies are retried with
|
||||
capped exponential backoff.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from ....components import R
|
||||
from ...base_step import BaseStep
|
||||
|
||||
DEFAULT_REFERENCE_PATHS = (
|
||||
"benchmark/longmemeval/golden_check_list_false.jsonl",
|
||||
"benchmark/longmemeval/merge_confirm_jinli_false.jsonl",
|
||||
)
|
||||
REFERENCE_PATHS_ENV = "LME_FINAL_ANSWER_REFERENCE_PATHS"
|
||||
RETRY_INITIAL_SECONDS = 5.0
|
||||
RETRY_MAX_SECONDS = 300.0
|
||||
_LME_DATETIME_RE = re.compile(r"(\d{4})/(\d{2})/(\d{2}).*?(\d{2}):(\d{2})")
|
||||
_FENCED_JSON_RE = re.compile(r"```json\s*(.*?)\s*```", re.IGNORECASE | re.DOTALL)
|
||||
|
||||
|
||||
@R.register("lme_final_answer_review_step")
|
||||
class FinalAnswerReviewStep(BaseStep):
|
||||
"""Ask a Claude Code agent to review one golden answer."""
|
||||
|
||||
@staticmethod
|
||||
def _load_json(path: Path) -> dict[str, Any]:
|
||||
try:
|
||||
with path.open(encoding="utf-8") as file:
|
||||
value = json.load(file)
|
||||
except OSError as exc:
|
||||
raise FileNotFoundError(f"Cannot read LongMemEval file: {path}") from exc
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError(f"Invalid JSON in LongMemEval file: {path}") from exc
|
||||
if not isinstance(value, dict):
|
||||
raise ValueError(f"Expected a JSON object in {path}")
|
||||
return value
|
||||
|
||||
@staticmethod
|
||||
def _parse_datetime(raw_date: Any, *, source: str) -> datetime:
|
||||
text = str(raw_date or "").strip()
|
||||
match = _LME_DATETIME_RE.search(text)
|
||||
if match is None:
|
||||
raise ValueError(f"Invalid LongMemEval datetime in {source}: {text!r}")
|
||||
try:
|
||||
return datetime(*(int(part) for part in match.groups()))
|
||||
except ValueError as exc:
|
||||
raise ValueError(
|
||||
f"Invalid LongMemEval datetime in {source}: {text!r}",
|
||||
) from exc
|
||||
|
||||
def _resolve_reference_path(self, raw_path: str) -> Path:
|
||||
path = Path(raw_path).expanduser()
|
||||
if path.is_absolute():
|
||||
return path
|
||||
|
||||
# The configured defaults are repository-relative. Tests and custom
|
||||
# jobs may instead provide workspace-relative fixture paths.
|
||||
repository_path = Path.cwd() / path
|
||||
if repository_path.is_file():
|
||||
return repository_path
|
||||
return self.workspace_path / path
|
||||
|
||||
def _load_references(self, question_id: str) -> list[dict[str, Any]]:
|
||||
raw_paths: Any
|
||||
serialized_paths = os.environ.get(REFERENCE_PATHS_ENV)
|
||||
if serialized_paths:
|
||||
try:
|
||||
raw_paths = json.loads(serialized_paths)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError(f"{REFERENCE_PATHS_ENV} must be a JSON array of paths") from exc
|
||||
else:
|
||||
raw_paths = self.kwargs.get("reference_paths") or DEFAULT_REFERENCE_PATHS
|
||||
if isinstance(raw_paths, str):
|
||||
raw_paths = [raw_paths]
|
||||
if not isinstance(raw_paths, (list, tuple)) or not raw_paths:
|
||||
raise ValueError("reference_paths must contain at least one JSONL path")
|
||||
|
||||
references: list[dict[str, Any]] = []
|
||||
for raw_path in raw_paths:
|
||||
path = self._resolve_reference_path(str(raw_path))
|
||||
try:
|
||||
with path.open(encoding="utf-8") as file:
|
||||
for line_number, line in enumerate(file, start=1):
|
||||
if not line.strip():
|
||||
continue
|
||||
try:
|
||||
item = json.loads(line)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError(
|
||||
f"Invalid JSONL at {path}:{line_number}",
|
||||
) from exc
|
||||
if not isinstance(item, dict):
|
||||
raise ValueError(
|
||||
f"Expected a JSON object at {path}:{line_number}",
|
||||
)
|
||||
if str(item.get("question_id") or "") == question_id:
|
||||
references.append({"source": path.name, **item})
|
||||
except OSError as exc:
|
||||
raise FileNotFoundError(
|
||||
f"Cannot read reference-answer file: {path}",
|
||||
) from exc
|
||||
|
||||
return references
|
||||
|
||||
def _inspect_session_times(self, question_dt: datetime) -> tuple[int, list[dict[str, str]]]:
|
||||
"""Return the session count and timestamp-only metadata for future sessions."""
|
||||
resource_dir = self.app_context.app_config.resource_dir if self.app_context is not None else "session"
|
||||
session_dir = self.workspace_path / resource_dir
|
||||
if not session_dir.is_dir():
|
||||
raise FileNotFoundError(f"Session directory not found: {session_dir}")
|
||||
|
||||
session_paths = sorted(session_dir.glob("*.json"))
|
||||
future_sessions: list[dict[str, str]] = []
|
||||
for path in session_paths:
|
||||
session = self._load_json(path)
|
||||
session_id = str(session.get("haystack_session_id") or path.stem)
|
||||
session_date = str(session.get("haystack_date") or "").strip()
|
||||
session_dt = self._parse_datetime(
|
||||
session_date,
|
||||
source=f"{path}:haystack_date",
|
||||
)
|
||||
if session_dt > question_dt:
|
||||
future_sessions.append(
|
||||
{
|
||||
"session_id": session_id,
|
||||
"session_date": session_date,
|
||||
"session_file": path.name,
|
||||
},
|
||||
)
|
||||
return len(session_paths), future_sessions
|
||||
|
||||
@staticmethod
|
||||
def _parse_reply(raw_reply: Any) -> dict[str, Any]:
|
||||
if not isinstance(raw_reply, str) or not raw_reply.strip():
|
||||
raise ValueError("Agent returned an empty reply")
|
||||
json_blocks = _FENCED_JSON_RE.findall(raw_reply)
|
||||
if len(json_blocks) != 1:
|
||||
raise ValueError("Agent reply must contain exactly one fenced ```json``` block")
|
||||
try:
|
||||
value = json.loads(json_blocks[0].strip())
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError("Agent's fenced json block is not valid JSON") from exc
|
||||
if not isinstance(value, dict):
|
||||
raise ValueError("Agent reply must be a JSON object")
|
||||
if set(value) != {"reason", "golden_answer_correct", "answer", "is_session_time_wrong"}:
|
||||
raise ValueError(
|
||||
"Agent reply must contain exactly 'reason', 'golden_answer_correct', 'answer', "
|
||||
"and 'is_session_time_wrong'",
|
||||
)
|
||||
answer = value["answer"]
|
||||
reason = value["reason"]
|
||||
golden_answer_correct = value["golden_answer_correct"]
|
||||
is_session_time_wrong = value["is_session_time_wrong"]
|
||||
if not isinstance(reason, str) or not reason.strip():
|
||||
raise ValueError("Agent reply 'reason' must be a non-empty string")
|
||||
if "answer_session_ids" in reason.casefold():
|
||||
raise ValueError("Agent reply 'reason' must not evaluate answer_session_ids")
|
||||
if not isinstance(golden_answer_correct, bool):
|
||||
raise ValueError("Agent reply 'golden_answer_correct' must be a boolean")
|
||||
if not isinstance(answer, str):
|
||||
raise ValueError("Agent reply 'answer' must be a string")
|
||||
answer = answer.strip()
|
||||
if golden_answer_correct and answer:
|
||||
raise ValueError("Agent reply 'answer' must be empty when golden_answer_correct is true")
|
||||
if not golden_answer_correct and not answer:
|
||||
raise ValueError("Agent reply 'answer' must be non-empty when golden_answer_correct is false")
|
||||
if not isinstance(is_session_time_wrong, bool):
|
||||
raise ValueError("Agent reply 'is_session_time_wrong' must be a boolean")
|
||||
if is_session_time_wrong:
|
||||
raise ValueError("Agent reply 'is_session_time_wrong' is deprecated and must be false")
|
||||
return {
|
||||
"reason": reason.strip(),
|
||||
"golden_answer_correct": golden_answer_correct,
|
||||
"answer": answer,
|
||||
"is_session_time_wrong": is_session_time_wrong,
|
||||
}
|
||||
|
||||
async def execute(self):
|
||||
assert self.context is not None
|
||||
if self.agent_wrapper is None:
|
||||
raise ValueError("lme_final_answer_review_step requires agent_wrapper")
|
||||
|
||||
query = self._load_json(self.workspace_path / "query.json")
|
||||
golden = self._load_json(self.workspace_path / "answer.json")
|
||||
question_id = str(query.get("question_id") or "").strip()
|
||||
if not question_id:
|
||||
raise ValueError("query.json requires a non-empty 'question_id'")
|
||||
question_dt = self._parse_datetime(
|
||||
query.get("question_date"),
|
||||
source="query.json:question_date",
|
||||
)
|
||||
references = self._load_references(question_id)
|
||||
num_sessions, future_sessions = self._inspect_session_times(question_dt)
|
||||
|
||||
payload = {
|
||||
"query": query,
|
||||
"answer_json": golden,
|
||||
"reference_answers": references,
|
||||
"session_time_check": {
|
||||
"sessions_after_question_date": future_sessions,
|
||||
},
|
||||
}
|
||||
user_prompt = self.prompt_format(
|
||||
"user_message",
|
||||
question_id=question_id,
|
||||
question_date=str(query.get("question_date") or ""),
|
||||
num_sessions=num_sessions,
|
||||
num_future_sessions=len(future_sessions),
|
||||
num_references=len(references),
|
||||
payload_json=json.dumps(payload, ensure_ascii=False, indent=2),
|
||||
)
|
||||
|
||||
retry_initial_seconds = float(
|
||||
self.kwargs.get("retry_initial_seconds", RETRY_INITIAL_SECONDS),
|
||||
)
|
||||
retry_max_seconds = float(
|
||||
self.kwargs.get("retry_max_seconds", RETRY_MAX_SECONDS),
|
||||
)
|
||||
if retry_initial_seconds <= 0:
|
||||
retry_initial_seconds = RETRY_INITIAL_SECONDS
|
||||
retry_max_seconds = max(retry_max_seconds, retry_initial_seconds)
|
||||
|
||||
attempt = 1
|
||||
sleep_seconds = retry_initial_seconds
|
||||
while True:
|
||||
try:
|
||||
# Deliberately do not pass output_schema: this case evaluates an
|
||||
# ordinary Claude Code response and validates it afterward.
|
||||
result = await self.agent_wrapper.reply(
|
||||
user_prompt,
|
||||
system_prompt=self.get_prompt("system_prompt"),
|
||||
)
|
||||
final_answer = self._parse_reply(result.get("result"))
|
||||
if attempt > 1:
|
||||
self.logger.info(
|
||||
f"[{self.name}] recovered after {attempt} attempts",
|
||||
)
|
||||
break
|
||||
except Exception as exc: # noqa: BLE001 - agent/API/format failures share the retry contract
|
||||
delay = min(sleep_seconds, retry_max_seconds)
|
||||
self.logger.warning(
|
||||
f"[{self.name}] attempt {attempt} failed for {question_id}: {exc}; retrying in {delay:.1f}s",
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
sleep_seconds = min(sleep_seconds * 2, retry_max_seconds)
|
||||
attempt += 1
|
||||
|
||||
self.context.response.success = True
|
||||
self.context.response.answer = json.dumps(final_answer, ensure_ascii=False)
|
||||
self.context.response.metadata.update(
|
||||
{
|
||||
"question_id": question_id,
|
||||
"num_sessions": num_sessions,
|
||||
"num_future_sessions": len(future_sessions),
|
||||
"future_sessions": future_sessions,
|
||||
"num_reference_answers": len(references),
|
||||
"is_session_time_wrong": False,
|
||||
"attempts": attempt,
|
||||
"agent_session_id": result.get("session_id"),
|
||||
},
|
||||
)
|
||||
return self.context.response
|
||||
|
|
@ -1,53 +0,0 @@
|
|||
system_prompt: |
|
||||
你是 LongMemEval 答案的最终审核员。完整的 query.json、answer.json，以及零个或多个可能正确、
|
||||
也可能错误的参考答案已经放在用户消息的 input JSON 中，不需要去其他目录寻找这些输入。没有参考
|
||||
答案时，应直接根据原始 session 独立审核 answer.json，不能因为缺少争议记录就假定 golden 答案正确。
|
||||
|
||||
你的当前工作目录就是该问题的 session 目录。目录中的每个 JSON 文件都是一个完整原始聊天 session。
|
||||
原始 session 内容没有预先放进上下文；请主动使用 Read、Glob、Grep、Bash 等工具在当前目录自由检索，
|
||||
并阅读所有与问题可能相关的 session。不要修改或删除这些文件。
|
||||
|
||||
你的任务是独立判断最合理的答案。answer.json 和 reference_answers 都只是待核对的线索，不是事实，
|
||||
不能因为多个参考答案一致就直接采纳。必须综合全部聊天记录，仔细区分用户与 assistant 的陈述，处理
|
||||
时间、更新、冲突、计数、偏好和指代关系。
|
||||
|
||||
检索时必须始终检查每个文件中的 haystack_date：发生在 question_date 之后的 session 属于未来
|
||||
信息，绝对不能用其聊天内容推导正确答案或判断 answer.json 正确。即使未来 session 给出了非常直接、
|
||||
看似正确或与参考答案一致的信息，也必须忽略其内容，避免时间穿越。必须先仅根据 question_date 当时
|
||||
已经存在的 session 独立得出正确答案，再与 answer.json 比较；合法证据不足时，正确答案为 unknown。
|
||||
|
||||
`answer_session_ids` 不属于本次审核对象。不要检查其是否完整、相关、存在或晚于 question_date，也
|
||||
不得因其包含未来、无关或错误的 session ID 而把 golden answer 判错。`golden_answer_correct` 只由
|
||||
`answer.json` 中 `answer` 的内容是否完整、正确决定。
|
||||
|
||||
input JSON 中的 session_time_check 只用于指出哪些 session 内容晚于 question_date、不能作为答题
|
||||
证据；它不用于检查 `answer_session_ids`。reason 中不需要评价 `answer_session_ids`。
|
||||
|
||||
你可以在最终回复中补充必要的分析文字，但必须包含且只能包含一个 ```json 代码块。程序只解析这个
|
||||
代码块；没有代码块、存在多个 json 代码块或块内 JSON 无效都会触发重试。代码块内必须是一个对象，
|
||||
且只能包含四个字段：
|
||||
- reason：中文详细推理。说明如何处理不同线索和参考答案，尽量逐条引用有证据作用的 session id、
|
||||
session 时间与具体事实，使后续人工 reviewer 可以复核。
|
||||
- golden_answer_correct：JSON boolean。仅根据 question_date 之前（含同一时刻）的 session 判断
|
||||
answer.json 中的 answer 是否完整且正确；不要考虑 answer_session_ids。
|
||||
- answer：仅当 golden_answer_correct 为 false 时，填写合法证据支持的正确答案（证据不足填
|
||||
unknown）；为 true 时必须填空字符串。
|
||||
- is_session_time_wrong：为兼容现有输出结构保留的弃用字段，始终填 false。
|
||||
|
||||
不要在 reason 或任何字段中评价 answer_session_ids。
|
||||
|
||||
输出格式示例仅用于说明 JSON 外形，不是内容 few-shot：
|
||||
```json
|
||||
{"reason":"详细推理与 session 证据","golden_answer_correct":false,"answer":"修正答案","is_session_time_wrong":false}
|
||||
```
|
||||
|
||||
user_message: |
|
||||
请审核 question_id={question_id}。
|
||||
Question date: {question_date}
|
||||
Session files in current working directory: {num_sessions}
|
||||
Sessions after question_date: {num_future_sessions}
|
||||
Reference answer count: {num_references}
|
||||
|
||||
以下 input JSON 包含完整 query.json、answer.json、参考答案和 session 时间检查结果。请先读完，再使用当前
|
||||
session 目录中的原始文件查找证据，独立推理后严格按 system prompt 要求输出带 ```json 代码块的结果：
|
||||
{payload_json}
|
||||
|
|
@ -1,188 +0,0 @@
|
|||
"""Judge whether the LongMemEval golden answer is reasonable.
|
||||
|
||||
Consumes ``session_review.json`` produced by ``lme_session_review_step`` and
|
||||
hands its extracted session information to an agent that is equipped with the
|
||||
``python_execute`` tool. The agent uses ``python_execute`` only as a scratchpad
|
||||
for checking the golden answer; ``answer_session_ids`` are outside the audit
|
||||
scope. The final verdict is not the
|
||||
raw Python stdout but a *structured* object extracted from the whole conversation
|
||||
via ``output_schema``. Sessions dated after ``question_date`` are filtered
|
||||
upstream by ``lme_session_review_step`` and are not included in this
|
||||
golden-check flow.
|
||||
|
||||
The output ``check_golden.json`` is intentionally slim: it does NOT duplicate the
|
||||
query/golden/review fields already stored in ``session_review.json`` (referenced by
|
||||
path), keeping only the relevant per-session ``session_summaries`` and the
|
||||
structured verdict. It is written to the workspace root (e.g.
|
||||
``datasets/longmemeval/1/check_golden.json``).
|
||||
"""
|
||||
|
||||
import json
|
||||
import asyncio
|
||||
from uuid import uuid4
|
||||
|
||||
from ...base_step import BaseStep
|
||||
from ....components import R
|
||||
|
||||
# File written under the workspace root with the full review + verdict payload.
|
||||
OUTPUT_FILENAME = "check_golden.json"
|
||||
SESSION_REVIEW_FILENAME = "session_review.json"
|
||||
RETRY_INITIAL_SECONDS = 5.0
|
||||
RETRY_MAX_SECONDS = 300.0
|
||||
|
||||
# Structured verdict the judge agent must produce (extracted from its reasoning).
|
||||
_VERDICT_SCHEMA = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"reasoning": {
|
||||
"type": "string",
|
||||
"description": "用中文写出详细的推理过程：先说明证据支持的答案，再判断 " "golden_answer 是否正确。",
|
||||
},
|
||||
"golden_answer_correct": {
|
||||
"type": "boolean",
|
||||
"description": "golden_answer 是否正确。",
|
||||
},
|
||||
"true_answer": {
|
||||
"type": "string",
|
||||
"description": "仅当 golden_answer_correct 为 false 时填写：证据支持的正确答案"
|
||||
"（证据不足时填 'unknown'）。golden_answer_correct 为 true 时填空字符串。",
|
||||
},
|
||||
},
|
||||
"required": [
|
||||
"reasoning",
|
||||
"golden_answer_correct",
|
||||
"true_answer",
|
||||
],
|
||||
"additionalProperties": False,
|
||||
}
|
||||
|
||||
|
||||
@R.register("lme_golden_check_step")
|
||||
class GoldenCheckStep(BaseStep):
|
||||
"""Let a python-enabled agent decide whether the golden answer holds up."""
|
||||
|
||||
@staticmethod
|
||||
def _compact_summary(summary: dict) -> dict:
|
||||
"""Keep only the evidence fields used by the golden-check prompt."""
|
||||
return {
|
||||
"session_id": str(summary.get("session_id") or ""),
|
||||
"session_date": str(summary.get("session_date") or ""),
|
||||
"extracted_info": str(summary.get("extracted_info") or ""),
|
||||
}
|
||||
|
||||
async def execute(self):
|
||||
assert self.context is not None
|
||||
if self.agent_wrapper is None:
|
||||
raise ValueError("lme_golden_check_step requires agent_wrapper")
|
||||
|
||||
review_path = self.workspace_path / SESSION_REVIEW_FILENAME
|
||||
if not review_path.is_file():
|
||||
raise FileNotFoundError(
|
||||
f"{SESSION_REVIEW_FILENAME} not found at {review_path}; run lme_session_review_step first",
|
||||
)
|
||||
try:
|
||||
with review_path.open(encoding="utf-8") as f:
|
||||
review_payload = json.load(f)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError(f"Invalid JSON in {review_path}") from exc
|
||||
if not isinstance(review_payload, dict):
|
||||
raise ValueError(f"Expected a JSON object in {review_path}")
|
||||
|
||||
query = review_payload.get("query") or {}
|
||||
golden = review_payload.get("golden") or {}
|
||||
# session_review.json keeps one extraction per reviewed session.
|
||||
session_summaries = [self._compact_summary(s) for s in review_payload.get("session_summaries") or []]
|
||||
|
||||
question = str(query.get("question") or "").strip()
|
||||
question_type = str(query.get("question_type") or "").strip()
|
||||
question_date = str(query.get("question_date") or "").strip()
|
||||
golden_answer = str(golden.get("answer") or "").strip()
|
||||
if not question:
|
||||
raise ValueError(f"{review_path} does not contain a question")
|
||||
|
||||
prompt_input = {
|
||||
"question": question,
|
||||
"question_type": question_type,
|
||||
"question_date": question_date,
|
||||
"golden_answer": golden_answer,
|
||||
"session_summaries": session_summaries,
|
||||
}
|
||||
user_prompt = self.prompt_format(
|
||||
"user_message",
|
||||
question=question,
|
||||
question_type=question_type,
|
||||
question_date=question_date,
|
||||
golden_answer=golden_answer,
|
||||
num_session_summaries=len(session_summaries),
|
||||
payload_json=json.dumps(prompt_input, ensure_ascii=False, indent=2),
|
||||
)
|
||||
|
||||
tool_context_id = str(self.context.get("tool_context_id") or f"lme-golden-{uuid4()}")
|
||||
retry_initial_seconds = float(self.kwargs.get("retry_initial_seconds", RETRY_INITIAL_SECONDS))
|
||||
retry_max_seconds = float(self.kwargs.get("retry_max_seconds", RETRY_MAX_SECONDS))
|
||||
retry_max_attempts_raw = self.kwargs.get("retry_max_attempts")
|
||||
retry_max_attempts = int(retry_max_attempts_raw) if retry_max_attempts_raw not in (None, "") else 0
|
||||
if retry_initial_seconds <= 0:
|
||||
retry_initial_seconds = RETRY_INITIAL_SECONDS
|
||||
retry_max_seconds = max(retry_max_seconds, retry_initial_seconds)
|
||||
|
||||
attempt = 1
|
||||
sleep_seconds = retry_initial_seconds
|
||||
while True:
|
||||
try:
|
||||
result = await self.agent_wrapper.reply(
|
||||
user_prompt,
|
||||
system_prompt=self.get_prompt("system_prompt"),
|
||||
tool_context_id=tool_context_id,
|
||||
output_schema=_VERDICT_SCHEMA,
|
||||
)
|
||||
if attempt > 1:
|
||||
self.logger.info(f"[{self.name}] golden check recovered after {attempt} attempts")
|
||||
break
|
||||
except Exception as exc:
|
||||
if 0 < retry_max_attempts <= attempt:
|
||||
raise
|
||||
next_sleep = min(sleep_seconds, retry_max_seconds)
|
||||
self.logger.warning(
|
||||
f"[{self.name}] golden check attempt {attempt} failed: {exc}; " f"retrying in {next_sleep:.1f}s",
|
||||
)
|
||||
await asyncio.sleep(next_sleep)
|
||||
sleep_seconds = min(sleep_seconds * 2, retry_max_seconds)
|
||||
attempt += 1
|
||||
|
||||
# The structured verdict is the real output; the free-text reply is only the
|
||||
# agent's closing narration and is kept as a fallback.
|
||||
verdict = result.get("structured_output")
|
||||
if not isinstance(verdict, dict):
|
||||
self.logger.warning(f"[{self.name}] no structured verdict; falling back to free text")
|
||||
verdict = {"reasoning": (result.get("result") or "").strip()}
|
||||
# Retain the legacy fields for readers of existing check_golden.json
|
||||
# artifacts. They are compatibility placeholders, not audit results.
|
||||
verdict["answer_session_ids_correct"] = True
|
||||
verdict["true_answer_session_ids"] = []
|
||||
|
||||
# Slim output: do NOT duplicate session_review.json (referenced by path);
|
||||
# keep only the compact session_summaries and the verdict.
|
||||
output = {
|
||||
"session_review_path": str(review_path),
|
||||
"session_summaries": session_summaries,
|
||||
"verdict": verdict,
|
||||
}
|
||||
output_path = self.workspace_path / OUTPUT_FILENAME
|
||||
with output_path.open("w", encoding="utf-8") as f:
|
||||
json.dump(output, f, ensure_ascii=False, indent=2)
|
||||
self.logger.info(f"[{self.name}] wrote verdict to {output_path}")
|
||||
|
||||
self.context.response.success = True
|
||||
self.context.response.answer = json.dumps(verdict, ensure_ascii=False, indent=2)
|
||||
self.context.response.metadata.update(
|
||||
{
|
||||
"num_session_summaries": len(session_summaries),
|
||||
"session_review_path": str(review_path),
|
||||
"tool_context_id": tool_context_id,
|
||||
"agent_session_id": result.get("session_id"),
|
||||
"output_path": str(output_path),
|
||||
"verdict": verdict,
|
||||
},
|
||||
)
|
||||
return self.context.response
|
||||
|
|
@ -1,27 +0,0 @@
|
|||
system_prompt: |
|
||||
你是 LongMemEval 基准测试的审核员。你要根据从用户聊天记录中提取的证据，判断某个问题的
|
||||
golden_answer 是否正确。只审核答案内容，不检查或评价 answer_session_ids。
|
||||
|
||||
你可以使用 python_execute 工具作为推理草稿本：统计相关会话、抽取答案。把给定的数据以字面量
|
||||
形式直接嵌入 Python 代码，使计算可复现；把中间结果以 JSON 打印出来，便于审计。
|
||||
|
||||
python 的 stdout 不是你的最终答案，只是草稿。计算充分、确信之后，停止调用 python，用中文
|
||||
给出结论。最终的结构化结果会从整段对话中自动抽取，所以务必把推理和结论清楚表达。
|
||||
|
||||
结构化输出要求：
|
||||
- reasoning 必须是详细的中文推理过程。
|
||||
- true_answer 仅在 golden_answer_correct 为 false 时填写，为空字符串否则。
|
||||
|
||||
user_message: |
|
||||
Question: {question}
|
||||
Question type: {question_type}
|
||||
Question date: {question_date}
|
||||
Golden answer: {golden_answer}
|
||||
Number of session extractions included: {num_session_summaries}
|
||||
|
||||
证据（JSON）。session_summaries 含上游审核过的会话，每条只有 session_id、session_date、
|
||||
extracted_info：
|
||||
{payload_json}
|
||||
|
||||
用 python_execute 统计和推理，过程中打印中间 JSON。然后用中文给出最终结论：golden_answer
|
||||
是否正确（不正确时给出 true_answer）。不要评价 answer_session_ids。
|
||||
81
reme/steps/benchmark/lme/llm_judge.py
Normal file
81
reme/steps/benchmark/lme/llm_judge.py
Normal file
|
|
@ -0,0 +1,81 @@
|
|||
"""Judge whether an agent answer matches the golden answer."""
|
||||
|
||||
import re
|
||||
|
||||
from ...base_step import BaseStep
|
||||
from ....components import R
|
||||
|
||||
|
||||
@R.register("lme_answer_judge_step")
|
||||
class LmeAnswerJudgeStep(BaseStep):
|
||||
"""Evaluate whether an agent answer is correct against a golden answer."""
|
||||
|
||||
PROMPT_KEYS_BY_QUESTION_TYPE = {
|
||||
"temporal_reasoning": "temporal_reasoning_system_prompt",
|
||||
"knowledge_update": "knowledge_update_system_prompt",
|
||||
"single_session_preference": "single_session_preference_system_prompt",
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def _judge_prompt_key(cls, question_type: str) -> str:
|
||||
normalized = question_type.strip().lower().replace("-", "_").replace(" ", "_")
|
||||
return cls.PROMPT_KEYS_BY_QUESTION_TYPE.get(normalized, "other_question_types_system_prompt")
|
||||
|
||||
@staticmethod
|
||||
def _normalize_judgement(raw_answer: str) -> str:
|
||||
match = re.match(r"\s*(yes|no)\b", raw_answer, re.IGNORECASE)
|
||||
if match:
|
||||
return match.group(1).lower()
|
||||
return raw_answer.strip().lower()
|
||||
|
||||
async def execute(self):
|
||||
assert self.context is not None
|
||||
query: str = self.context.get("query", "")
|
||||
agent_answer: str = self.context.get("agent_answer", "")
|
||||
golden_answer: str = self.context.get("golden_answer", "")
|
||||
question_type: str = self.context.get("question_type", "")
|
||||
|
||||
if not query:
|
||||
raise ValueError("lme_answer_judge_step requires non-empty query")
|
||||
if not agent_answer:
|
||||
raise ValueError("lme_answer_judge_step requires non-empty agent_answer")
|
||||
if not golden_answer:
|
||||
raise ValueError("lme_answer_judge_step requires non-empty golden_answer")
|
||||
if self.agent_wrapper is None:
|
||||
raise RuntimeError("lme_answer_judge_step requires agent_wrapper")
|
||||
|
||||
judge_prompt_key = self._judge_prompt_key(question_type)
|
||||
user_prompt_key = (
|
||||
"preference_judge_user_message"
|
||||
if judge_prompt_key == "single_session_preference_system_prompt"
|
||||
else "answer_judge_user_message"
|
||||
)
|
||||
user_prompt = self.prompt_format(
|
||||
user_prompt_key,
|
||||
query=query,
|
||||
golden_answer=golden_answer,
|
||||
agent_answer=agent_answer,
|
||||
)
|
||||
result = await self.agent_wrapper.reply(
|
||||
user_prompt,
|
||||
system_prompt=self.prompt_format(judge_prompt_key),
|
||||
)
|
||||
|
||||
raw_answer = (result.get("result") or "").strip()
|
||||
answer = self._normalize_judgement(raw_answer)
|
||||
|
||||
self.logger.info(f"[{self.name}] answer judgement: {answer}")
|
||||
self.context["answer_judgement"] = answer
|
||||
self.context.response.success = True
|
||||
self.context.response.answer = answer
|
||||
self.context.response.metadata.update(
|
||||
{
|
||||
"query": query,
|
||||
"agent_answer": agent_answer,
|
||||
"golden_answer": golden_answer,
|
||||
"question_type": question_type,
|
||||
"answer_judgement": answer,
|
||||
"raw_answer_judgement": raw_answer,
|
||||
},
|
||||
)
|
||||
return self.context.response
|
||||
|
|
@ -1,109 +0,0 @@
|
|||
"""LME file-based LLM judge step."""
|
||||
|
||||
import json
|
||||
import re
|
||||
|
||||
from ...base_step import BaseStep
|
||||
from ....components import R
|
||||
|
||||
|
||||
@R.register("lme_llm_judge_step")
|
||||
class LmeLlmJudgeStep(BaseStep):
|
||||
"""Judge ``mem_answer.json`` against ``answer.json`` and update it in place."""
|
||||
|
||||
PROMPT_KEYS_BY_QUESTION_TYPE = {
|
||||
"temporal_reasoning": "temporal_reasoning_system_prompt",
|
||||
"knowledge_update": "knowledge_update_system_prompt",
|
||||
"single_session_preference": "single_session_preference_system_prompt",
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def _judge_prompt_key(cls, question_type: str) -> str:
|
||||
normalized = question_type.strip().lower().replace("-", "_").replace(" ", "_")
|
||||
return cls.PROMPT_KEYS_BY_QUESTION_TYPE.get(normalized, "other_question_types_system_prompt")
|
||||
|
||||
@staticmethod
|
||||
def _user_prompt_key(judge_prompt_key: str) -> str:
|
||||
if judge_prompt_key == "single_session_preference_system_prompt":
|
||||
return "preference_judge_user_message"
|
||||
return "answer_judge_user_message"
|
||||
|
||||
@staticmethod
|
||||
def _normalize_judgement(raw_answer: str) -> str:
|
||||
match = re.match(r"\s*(yes|no)\b", raw_answer, re.IGNORECASE)
|
||||
if match:
|
||||
return match.group(1).lower()
|
||||
return raw_answer.strip().lower()
|
||||
|
||||
def _load_json_object(self, filename: str) -> dict:
|
||||
path = self.workspace_path / filename
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(f"{filename} does not exist in {self.workspace_path}")
|
||||
with path.open(encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError(f"{filename} is not a JSON object")
|
||||
return data
|
||||
|
||||
async def execute(self):
|
||||
assert self.context is not None
|
||||
if self.agent_wrapper is None:
|
||||
raise RuntimeError("lme_llm_judge_step requires agent_wrapper")
|
||||
|
||||
query_data = self._load_json_object("query.json")
|
||||
golden_data = self._load_json_object("answer.json")
|
||||
mem_answer = self._load_json_object("mem_answer.json")
|
||||
|
||||
query = str(query_data.get("question", "") or "").strip()
|
||||
agent_answer = str(mem_answer.get("answer", "") or "").strip()
|
||||
golden_answer = str(golden_data.get("answer", "") or "").strip()
|
||||
question_type = str(query_data.get("question_type", "") or "")
|
||||
|
||||
if not query:
|
||||
raise ValueError("query.json requires a non-empty 'question'")
|
||||
if not agent_answer:
|
||||
raise ValueError("mem_answer.json requires a non-empty 'answer'")
|
||||
if not golden_answer:
|
||||
raise ValueError("answer.json requires a non-empty 'answer'")
|
||||
|
||||
judge_prompt_key = self._judge_prompt_key(question_type)
|
||||
user_prompt = self.prompt_format(
|
||||
self._user_prompt_key(judge_prompt_key),
|
||||
query=query,
|
||||
golden_answer=golden_answer,
|
||||
agent_answer=agent_answer,
|
||||
)
|
||||
result = await self.agent_wrapper.reply(
|
||||
user_prompt,
|
||||
system_prompt=self.prompt_format(judge_prompt_key),
|
||||
)
|
||||
|
||||
raw_answer = (result.get("result") or "").strip()
|
||||
answer = self._normalize_judgement(raw_answer)
|
||||
|
||||
mem_answer["llm_judge"] = {
|
||||
"judgement": answer,
|
||||
"raw_judgement": raw_answer,
|
||||
"golden_answer": golden_answer,
|
||||
"question_type": question_type,
|
||||
}
|
||||
out_path = self.workspace_path / "mem_answer.json"
|
||||
out_path.write_text(json.dumps(mem_answer, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
question_id = str(query_data.get("question_id") or mem_answer.get("question_id") or "")
|
||||
self.logger.info(f"[{self.name}] llm judgement for {question_id or query!r}: {answer}")
|
||||
self.context.response.success = True
|
||||
self.context.response.answer = answer
|
||||
self.context.response.metadata.update(
|
||||
{
|
||||
"question_id": question_id,
|
||||
"query": query,
|
||||
"agent_answer": agent_answer,
|
||||
"golden_answer": golden_answer,
|
||||
"question_type": question_type,
|
||||
"answer_judgement": answer,
|
||||
"raw_answer_judgement": raw_answer,
|
||||
"path": "mem_answer.json",
|
||||
},
|
||||
)
|
||||
return self.context.response
|
||||
|
|
@ -1,358 +0,0 @@
|
|||
"""Review every LongMemEval session and extract its information.
|
||||
|
||||
For a workspace such as ``datasets/longmemeval/1`` this step loads ``query.json``
|
||||
and ``answer.json``, filters out sessions dated after ``question_date``, then
|
||||
walks each remaining session under ``resource_dir`` one by one. An agent wrapper
|
||||
extracts the complete information in each session, with extra care not to omit
|
||||
anything related to the question or golden answer.
|
||||
|
||||
The collected per-session extractions are written to ``session_review.json`` for
|
||||
the downstream golden-answer check.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from ...base_step import BaseStep
|
||||
from ....components import R
|
||||
|
||||
START_INTERVAL_SECONDS = 1.0
|
||||
MAX_CONCURRENCY = 60
|
||||
RETRY_INITIAL_SECONDS = 5.0
|
||||
RETRY_MAX_SECONDS = 300.0
|
||||
OUTPUT_FILENAME = "session_review.json"
|
||||
_LME_DATETIME_RE = re.compile(r"(\d{4})/(\d{2})/(\d{2}).*?(\d{2}):(\d{2})")
|
||||
_NON_RETRYABLE_DATA_INSPECTION_MARKERS = (
|
||||
"data_inspection_failed",
|
||||
"DataInspectionFailed",
|
||||
"Input text data may contain inappropriate content",
|
||||
)
|
||||
|
||||
|
||||
@R.register("lme_session_review_step")
|
||||
class SessionReviewStep(BaseStep):
|
||||
"""Extract complete information from every eligible session."""
|
||||
|
||||
def _load_json(self, path: Path | str) -> dict:
|
||||
if not isinstance(path, Path):
|
||||
path = self.workspace_path / path
|
||||
try:
|
||||
with path.open(encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
except OSError as exc:
|
||||
raise FileNotFoundError(f"Cannot read LongMemEval file: {path}") from exc
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError(f"Invalid JSON in LongMemEval file: {path}") from exc
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError(f"Expected a JSON object in {path}")
|
||||
return data
|
||||
|
||||
def _session_dir(self) -> Path:
|
||||
resource_dir = self.app_context.app_config.resource_dir if self.app_context is not None else "session"
|
||||
return self.workspace_path / resource_dir
|
||||
|
||||
@staticmethod
|
||||
def _parse_lme_datetime(raw_date: str) -> datetime | None:
|
||||
"""Parse LongMemEval timestamps like ``2023/05/20 (Sat) 03:29``."""
|
||||
match = _LME_DATETIME_RE.search(raw_date.strip())
|
||||
if match is None:
|
||||
return None
|
||||
try:
|
||||
year, month, day, hour, minute = (int(part) for part in match.groups())
|
||||
return datetime(year, month, day, hour, minute)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _is_data_inspection_error(exc: Exception) -> bool:
|
||||
text = str(exc)
|
||||
return any(marker in text for marker in _NON_RETRYABLE_DATA_INSPECTION_MARKERS)
|
||||
|
||||
# pylint: disable=too-many-statements
|
||||
async def execute(self):
|
||||
assert self.context is not None
|
||||
if self.agent_wrapper is None:
|
||||
raise ValueError("lme_session_review_step requires agent_wrapper")
|
||||
|
||||
query_data = self._load_json("query.json")
|
||||
answer_data = self._load_json("answer.json")
|
||||
|
||||
question = str(query_data.get("question") or "").strip()
|
||||
question_type = str(query_data.get("question_type") or "").strip()
|
||||
question_date = str(query_data.get("question_date") or "").strip()
|
||||
if not question:
|
||||
raise ValueError("query.json requires a non-empty 'question'")
|
||||
question_dt = self._parse_lme_datetime(question_date)
|
||||
if question_dt is None:
|
||||
raise ValueError(f"query.json has an invalid 'question_date': {question_date!r}")
|
||||
|
||||
golden_answer = str(answer_data.get("answer") or "").strip()
|
||||
answer_session_ids = [str(s) for s in (answer_data.get("answer_session_ids") or [])]
|
||||
|
||||
session_dir = self._session_dir()
|
||||
if not session_dir.is_dir():
|
||||
raise FileNotFoundError(f"Session directory not found: {session_dir}")
|
||||
session_files = sorted(p for p in session_dir.iterdir() if p.suffix == ".json")
|
||||
sessions: list[tuple[dict, str, str]] = []
|
||||
filtered_sessions: list[dict] = []
|
||||
session_ids_illegal: list[str] = []
|
||||
answer_session_ids_illegal: list[str] = []
|
||||
answer_session_id_set = set(answer_session_ids)
|
||||
|
||||
for session_path in session_files:
|
||||
try:
|
||||
session = self._load_json(session_path)
|
||||
except (ValueError, FileNotFoundError) as exc:
|
||||
self.logger.warning(f"[{self.name}] skip {session_path.name}: {exc}")
|
||||
continue
|
||||
|
||||
session_id = str(session.get("haystack_session_id") or session_path.stem)
|
||||
session_date = str(session.get("haystack_date") or "").strip()
|
||||
session_dt = self._parse_lme_datetime(session_date)
|
||||
if session_dt is not None and session_dt > question_dt:
|
||||
session_ids_illegal.append(session_id)
|
||||
filtered_sessions.append(
|
||||
{
|
||||
"session_id": session_id,
|
||||
"session_date": session_date,
|
||||
"session_file": session_path.name,
|
||||
"reason": "session_date_after_question_date",
|
||||
},
|
||||
)
|
||||
if session_id in answer_session_id_set:
|
||||
answer_session_ids_illegal.append(session_id)
|
||||
continue
|
||||
if session_dt is None:
|
||||
self.logger.warning(
|
||||
f"[{self.name}] keep {session_id}: cannot parse haystack_date={session_date!r}",
|
||||
)
|
||||
sessions.append((session, session_id, session_date))
|
||||
|
||||
illegal_answer_session_ids = set(answer_session_ids_illegal)
|
||||
answer_session_ids_filter_illegal = [
|
||||
session_id for session_id in answer_session_ids if session_id not in illegal_answer_session_ids
|
||||
]
|
||||
total = len(sessions)
|
||||
start_interval_seconds = float(self.kwargs.get("start_interval_seconds", START_INTERVAL_SECONDS))
|
||||
if start_interval_seconds < 0:
|
||||
start_interval_seconds = START_INTERVAL_SECONDS
|
||||
concurrency = int(self.kwargs.get("concurrency", MAX_CONCURRENCY))
|
||||
if concurrency <= 0:
|
||||
concurrency = MAX_CONCURRENCY
|
||||
concurrency = min(concurrency, MAX_CONCURRENCY)
|
||||
self.logger.info(
|
||||
f"[{self.name}] reviewing {total} sessions from {session_dir} "
|
||||
f"(filtered {len(session_ids_illegal)} sessions after question_date, "
|
||||
f"start_interval={start_interval_seconds}s, concurrency={concurrency})",
|
||||
)
|
||||
|
||||
failed_reviews: list[dict] = []
|
||||
retry_initial_seconds = float(self.kwargs.get("retry_initial_seconds", RETRY_INITIAL_SECONDS))
|
||||
retry_max_seconds = float(self.kwargs.get("retry_max_seconds", RETRY_MAX_SECONDS))
|
||||
retry_max_attempts_raw = self.kwargs.get("retry_max_attempts")
|
||||
retry_max_attempts = int(retry_max_attempts_raw) if retry_max_attempts_raw not in (None, "") else 0
|
||||
if retry_initial_seconds <= 0:
|
||||
retry_initial_seconds = RETRY_INITIAL_SECONDS
|
||||
retry_max_seconds = max(retry_max_seconds, retry_initial_seconds)
|
||||
retry_gate = asyncio.Condition()
|
||||
retry_sleeping_review_idxs: set[int] = set()
|
||||
submit_lock = asyncio.Lock()
|
||||
last_submitted_at = 0.0
|
||||
|
||||
def has_prior_retry_sleeping(idx: int) -> bool:
|
||||
return any(retry_idx < idx for retry_idx in retry_sleeping_review_idxs)
|
||||
|
||||
async def wait_for_start_slot() -> None:
|
||||
nonlocal last_submitted_at
|
||||
async with submit_lock:
|
||||
sleep_seconds = last_submitted_at + start_interval_seconds - time.monotonic()
|
||||
if sleep_seconds > 0:
|
||||
await asyncio.sleep(sleep_seconds)
|
||||
last_submitted_at = time.monotonic()
|
||||
|
||||
async def wait_for_healthy_start_slot(idx: int, session_id: str) -> None:
|
||||
while True:
|
||||
async with retry_gate:
|
||||
if has_prior_retry_sleeping(idx):
|
||||
self.logger.info(
|
||||
f"[{self.name}] ({idx}/{total}) {session_id} waits for earlier retry sleep",
|
||||
)
|
||||
await retry_gate.wait_for(lambda: not has_prior_retry_sleeping(idx))
|
||||
|
||||
await wait_for_start_slot()
|
||||
|
||||
async with retry_gate:
|
||||
if not has_prior_retry_sleeping(idx):
|
||||
return
|
||||
|
||||
async def mark_retry_sleeping(idx: int) -> None:
|
||||
async with retry_gate:
|
||||
retry_sleeping_review_idxs.add(idx)
|
||||
retry_gate.notify_all()
|
||||
|
||||
async def mark_retry_awake(idx: int) -> None:
|
||||
async with retry_gate:
|
||||
retry_sleeping_review_idxs.discard(idx)
|
||||
retry_gate.notify_all()
|
||||
|
||||
async def reply_with_retry(idx: int, user_prompt: str, session_id: str) -> dict:
|
||||
attempt = 1
|
||||
sleep_seconds = retry_initial_seconds
|
||||
while True:
|
||||
try:
|
||||
await wait_for_healthy_start_slot(idx, session_id)
|
||||
result = await self.agent_wrapper.reply(
|
||||
user_prompt,
|
||||
system_prompt=self.get_prompt("system_prompt"),
|
||||
)
|
||||
await mark_retry_awake(idx)
|
||||
if attempt > 1:
|
||||
self.logger.info(f"[{self.name}] review recovered for {session_id} after {attempt} attempts")
|
||||
return result
|
||||
except Exception as exc:
|
||||
if self._is_data_inspection_error(exc):
|
||||
await mark_retry_awake(idx)
|
||||
raise
|
||||
if 0 < retry_max_attempts <= attempt:
|
||||
await mark_retry_awake(idx)
|
||||
raise
|
||||
await mark_retry_sleeping(idx)
|
||||
next_sleep = min(sleep_seconds, retry_max_seconds)
|
||||
self.logger.warning(
|
||||
f"[{self.name}] review attempt {attempt} failed for {session_id}: {exc}; "
|
||||
f"retrying in {next_sleep:.1f}s",
|
||||
)
|
||||
await asyncio.sleep(next_sleep)
|
||||
await mark_retry_awake(idx)
|
||||
sleep_seconds = min(sleep_seconds * 2, retry_max_seconds)
|
||||
attempt += 1
|
||||
|
||||
async def review_one(idx: int, session: dict, session_id: str, session_date: str) -> dict | None:
|
||||
user_prompt = self.prompt_format(
|
||||
"user_message",
|
||||
question=question,
|
||||
question_type=question_type,
|
||||
question_date=question_date,
|
||||
golden_answer=golden_answer,
|
||||
session_id=session_id,
|
||||
session_date=session_date,
|
||||
session_content=json.dumps(session.get("messages", []), ensure_ascii=False, indent=2),
|
||||
)
|
||||
try:
|
||||
result = await reply_with_retry(idx, user_prompt, session_id)
|
||||
except Exception as exc: # noqa: BLE001 — one bad session must not abort the sweep
|
||||
if self._is_data_inspection_error(exc):
|
||||
error = str(exc)
|
||||
self.logger.warning(
|
||||
f"[{self.name}] review fallback for {session_id}: non-retryable data inspection error",
|
||||
)
|
||||
failed_reviews.append(
|
||||
{
|
||||
"session_id": session_id,
|
||||
"session_date": session_date,
|
||||
"error": error,
|
||||
"non_retryable": True,
|
||||
"fallback": True,
|
||||
"fallback_reason": "data_inspection_failed",
|
||||
"raw_session": session,
|
||||
},
|
||||
)
|
||||
return {
|
||||
"session_id": session_id,
|
||||
"session_date": session_date,
|
||||
"extracted_info": "",
|
||||
"review_status": "fallback",
|
||||
"fallback_reason": "data_inspection_failed",
|
||||
"error": error,
|
||||
"raw_session": session,
|
||||
}
|
||||
self.logger.warning(f"[{self.name}] review failed for {session_id}: {exc}")
|
||||
failed_reviews.append(
|
||||
{
|
||||
"session_id": session_id,
|
||||
"session_date": session_date,
|
||||
"error": str(exc),
|
||||
"non_retryable": False,
|
||||
"fallback": False,
|
||||
},
|
||||
)
|
||||
return None
|
||||
|
||||
extracted_info = str(result.get("result") or "").strip()
|
||||
|
||||
summary = {
|
||||
"session_id": session_id,
|
||||
"session_date": session_date,
|
||||
"extracted_info": extracted_info,
|
||||
}
|
||||
self.logger.info(f"[{self.name}] ({idx}/{total}) extracted {session_id}")
|
||||
return summary
|
||||
|
||||
review_semaphore = asyncio.Semaphore(concurrency)
|
||||
|
||||
async def review_one_limited(idx: int, session: dict, session_id: str, session_date: str) -> dict | None:
|
||||
async with review_semaphore:
|
||||
return await review_one(idx, session, session_id, session_date)
|
||||
|
||||
# gather preserves input order, so summaries stay chronological.
|
||||
results = await asyncio.gather(
|
||||
*(
|
||||
review_one_limited(idx, session, session_id, session_date)
|
||||
for idx, (session, session_id, session_date) in enumerate(sessions, start=1)
|
||||
),
|
||||
)
|
||||
summaries: list[dict] = [s for s in results if s is not None]
|
||||
non_empty_summaries = [s for s in summaries if str(s.get("extracted_info") or "").strip()]
|
||||
fallback_summaries = [s for s in summaries if s.get("review_status") == "fallback"]
|
||||
reviewed_session_ids = [str(s.get("session_id")) for s in summaries if s.get("session_id")]
|
||||
output = {
|
||||
"query": {
|
||||
"question_id": query_data.get("question_id"),
|
||||
"question": question,
|
||||
"question_type": question_type,
|
||||
"question_date": question_date,
|
||||
},
|
||||
"golden": {
|
||||
"answer": golden_answer,
|
||||
"answer_session_ids": answer_session_ids,
|
||||
"answer_session_ids_filter_illegal": answer_session_ids_filter_illegal,
|
||||
"answer_session_ids_illegal": answer_session_ids_illegal,
|
||||
},
|
||||
"review": {
|
||||
"num_session_files": len(session_files),
|
||||
"num_reviewed_sessions": len(summaries),
|
||||
"num_extracted_sessions": len(non_empty_summaries),
|
||||
"num_empty_extractions": len(summaries) - len(non_empty_summaries),
|
||||
"num_failed_reviews": len(failed_reviews),
|
||||
"num_fallback_reviews": len(fallback_summaries),
|
||||
"num_filtered_sessions": len(session_ids_illegal),
|
||||
"reviewed_session_ids": reviewed_session_ids,
|
||||
"session_ids_illegal": session_ids_illegal,
|
||||
"filtered_sessions": filtered_sessions,
|
||||
"failed_reviews": failed_reviews,
|
||||
"fallback_reviews": fallback_summaries,
|
||||
},
|
||||
"session_summaries": summaries,
|
||||
}
|
||||
output_path = self.workspace_path / OUTPUT_FILENAME
|
||||
with output_path.open("w", encoding="utf-8") as f:
|
||||
json.dump(output, f, ensure_ascii=False, indent=2)
|
||||
self.logger.info(f"[{self.name}] wrote session review to {output_path}")
|
||||
|
||||
self.context.response.success = True
|
||||
self.context.response.answer = f"reviewed {len(summaries)} sessions"
|
||||
self.context.response.metadata.update(
|
||||
{
|
||||
"num_session_files": len(session_files),
|
||||
"num_reviewed_sessions": len(summaries),
|
||||
"num_failed_reviews": len(failed_reviews),
|
||||
"num_fallback_reviews": len(fallback_summaries),
|
||||
"num_filtered_sessions": len(session_ids_illegal),
|
||||
"output_path": str(output_path),
|
||||
},
|
||||
)
|
||||
return self.context.response
|
||||
|
|
@ -1,31 +0,0 @@
|
|||
system_prompt: |
|
||||
You are an information extractor for the LongMemEval benchmark. You are given ONE chat session,
|
||||
a target question, and its golden answer. Your job is to extract the complete information in
|
||||
this session, especially anything related to the question or golden answer.
|
||||
|
||||
Rules:
|
||||
- Extract all facts, names, entities, numbers, preferences, constraints, corrections, updates,
|
||||
contradictions, plans, events, and outcomes that appear in the session.
|
||||
- Be especially careful not to omit any information related to the question or golden answer.
|
||||
- Keep time expressions inline with the fact they modify, including dates, weekdays, relative
|
||||
times such as "last week" or "since January 15th", durations, and frequencies.
|
||||
- Do not invent facts. Only extract what is actually present in the session.
|
||||
- Do not judge whether answer_session_ids are correct. They are outside the answer audit scope.
|
||||
- Output only the extracted information as plain text. Do not output JSON, markdown fences, or
|
||||
relevance labels.
|
||||
|
||||
user_message: |
|
||||
Question: {question}
|
||||
Question type: {question_type}
|
||||
Question date: {question_date}
|
||||
Golden answer: {golden_answer}
|
||||
|
||||
--- Session under review ---
|
||||
Session id: {session_id}
|
||||
Session date: {session_date}
|
||||
Session messages (JSON):
|
||||
{session_content}
|
||||
--- End of session ---
|
||||
|
||||
Extract the complete information from this session. Do not summarize away details, and do not
|
||||
omit information related to the question or golden answer.
|
||||
|
|
@ -224,6 +224,22 @@ class AutoMemoryStep(BaseStep):
|
|||
days = [day for msg in messages if (day := extract_daily_date(msg.created_at))]
|
||||
return max(days) if days else None
|
||||
|
||||
def _build_messages(self, raw_messages: list) -> list[Msg]:
|
||||
"""Convert raw message payloads into ``Msg`` objects.
|
||||
|
||||
Overridable hook: subclasses can preprocess ``raw_messages`` (e.g. fill
|
||||
in missing timestamps) before conversion.
|
||||
"""
|
||||
return [self._to_msg(item) for item in raw_messages]
|
||||
|
||||
def _reply_extra_kwargs(self, day: str) -> dict: # pylint: disable=unused-argument
|
||||
"""Extra keyword arguments for ``agent_wrapper.reply``.
|
||||
|
||||
Overridable hook: subclasses can inject additional reply options such
|
||||
as per-tool defaults keyed on ``day``.
|
||||
"""
|
||||
return {}
|
||||
|
||||
# pylint: disable=too-many-return-statements
|
||||
async def execute(self):
|
||||
assert self.context is not None
|
||||
|
|
@ -234,7 +250,7 @@ class AutoMemoryStep(BaseStep):
|
|||
tz = self.app_context.app_config.timezone if self.app_context is not None else None
|
||||
current = now(tz)
|
||||
|
||||
messages: list[Msg] = [self._to_msg(item) for item in raw_messages]
|
||||
messages: list[Msg] = self._build_messages(raw_messages)
|
||||
self.logger.info(
|
||||
f"[{self.name}] start session_id={session_id!r} raw_messages={len(raw_messages)} "
|
||||
f"messages={len(messages)} hint={bool(memory_hint)}",
|
||||
|
|
@ -300,6 +316,7 @@ class AutoMemoryStep(BaseStep):
|
|||
user_message,
|
||||
system_prompt=self.prompt_format("system_prompt"),
|
||||
job_tools=self.create_tools if created else self.update_tools,
|
||||
**self._reply_extra_kwargs(day),
|
||||
)
|
||||
self.logger.info(f"[{self.name}] agent done path={note_path} has_result={bool(result.get('result'))}")
|
||||
|
||||
|
|
|
|||
|
|
@ -139,7 +139,21 @@ class SearchStep(BaseStep):
|
|||
query: str = (self.context.get("query", "") or "").strip()
|
||||
limit: int = int(self.context.get("limit") or _default_limit())
|
||||
min_score: float = float(self.context.get("min_score") or 0.0)
|
||||
vector_weight: float = float(self.kwargs.get("vector_weight", 0.7))
|
||||
# vector_weight: prefer agent-supplied context value; fallback to YAML kwargs / default 0.7.
|
||||
# Convertible numeric inputs are clipped to [0.0, 1.0]; non-numeric inputs are silently ignored.
|
||||
raw_vw = self.context.get("vector_weight")
|
||||
vector_weight: float | None = None
|
||||
if raw_vw is not None:
|
||||
try:
|
||||
vector_weight = float(raw_vw)
|
||||
except (TypeError, ValueError):
|
||||
self.logger.warning(
|
||||
f"[{self.name}] non-numeric vector_weight={raw_vw!r}; ignoring and using default 0.7",
|
||||
)
|
||||
vector_weight = None
|
||||
if vector_weight is None:
|
||||
vector_weight = float(self.kwargs.get("vector_weight", 0.7))
|
||||
vector_weight = max(0.0, min(1.0, vector_weight))
|
||||
candidate_multiplier: float = float(self.kwargs.get("candidate_multiplier", 5.0))
|
||||
expand_links_enabled: bool = bool(self.kwargs.get("expand_links", True))
|
||||
max_links_per_direction: int = int(self.kwargs.get("max_links_per_direction", 10))
|
||||
|
|
@ -152,7 +166,6 @@ class SearchStep(BaseStep):
|
|||
self.context.response.success = False
|
||||
self.context.response.answer = "Error: query cannot be empty"
|
||||
return self.context.response
|
||||
assert 0.0 <= vector_weight <= 1.0, f"vector_weight must be in [0, 1], got {vector_weight}"
|
||||
assert limit > 0, f"limit must be positive, got {limit}"
|
||||
|
||||
candidates = min(_MAX_CANDIDATES, max(1, int(limit * candidate_multiplier)))
|
||||
|
|
|
|||
|
|
@ -1,425 +0,0 @@
|
|||
"""Focused tests for the disputed LongMemEval final-answer workflow."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
import pytest
|
||||
|
||||
from benchmark.longmemeval import run_final_answer_review as driver_module
|
||||
from benchmark.longmemeval.run_final_answer_review import (
|
||||
REFERENCE_PATHS_ENV,
|
||||
atomic_write_results,
|
||||
merge_references,
|
||||
select_question_ids,
|
||||
)
|
||||
from reme.components.agent_wrapper.base_agent_wrapper import BaseAgentWrapper
|
||||
from reme.components.agent_wrapper.cc_agent_wrapper import CcAgentWrapper
|
||||
from reme.components.application_context import ApplicationContext
|
||||
from reme.config import resolve_app_config
|
||||
from reme.steps.benchmark.lme import final_answer_review as review_module
|
||||
from reme.steps.benchmark.lme.final_answer_review import FinalAnswerReviewStep
|
||||
|
||||
|
||||
class _FakeAgentWrapper(BaseAgentWrapper):
|
||||
"""Return queued ordinary text replies and retain every prompt call."""
|
||||
|
||||
def __init__(self, replies: list[str]):
|
||||
super().__init__()
|
||||
self.replies = list(replies)
|
||||
self.calls: list[tuple[str, dict]] = []
|
||||
|
||||
async def reply(self, inputs, **kwargs) -> dict:
|
||||
"""Return the next queued agent response."""
|
||||
self.calls.append((inputs, kwargs))
|
||||
return {
|
||||
"session_id": f"attempt-{len(self.calls)}",
|
||||
"result": self.replies.pop(0),
|
||||
}
|
||||
|
||||
|
||||
def _write_json(path: Path, value: object) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(value, ensure_ascii=False), encoding="utf-8")
|
||||
|
||||
|
||||
def _write_jsonl(path: Path, rows: list[dict]) -> None:
|
||||
path.write_text(
|
||||
"".join(json.dumps(row, ensure_ascii=False) + "\n" for row in rows),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def _session(session_id: str, date: str, marker: str) -> dict:
|
||||
return {
|
||||
"haystack_session_id": session_id,
|
||||
"haystack_date": date,
|
||||
"messages": [{"role": "user", "content": marker}],
|
||||
"other_session_field": f"full-{marker}",
|
||||
}
|
||||
|
||||
|
||||
def test_final_answer_review_keeps_raw_sessions_out_of_prompt_and_retries_plain_json(
|
||||
tmp_path,
|
||||
monkeypatch,
|
||||
):
|
||||
"""Raw session messages stay on disk, and invalid ordinary replies are retried."""
|
||||
query = {
|
||||
"question_id": "question-1",
|
||||
"question": "What happened?",
|
||||
"question_type": "single-session-user",
|
||||
"question_date": "2024/01/02 (Tue) 10:00",
|
||||
"extra_query_field": "keep-me",
|
||||
}
|
||||
golden = {
|
||||
"answer": "old answer",
|
||||
"answer_session_ids": ["past", "future"],
|
||||
"extra_answer_field": "keep-me-too",
|
||||
}
|
||||
_write_json(tmp_path / "query.json", query)
|
||||
_write_json(tmp_path / "answer.json", golden)
|
||||
_write_json(
|
||||
tmp_path / "session" / "past.json",
|
||||
_session("past", "2024/01/02 (Tue) 09:59", "past-evidence"),
|
||||
)
|
||||
_write_json(
|
||||
tmp_path / "session" / "equal.json",
|
||||
_session("equal", "2024/01/02 (Tue) 10:00", "equal-evidence"),
|
||||
)
|
||||
_write_json(
|
||||
tmp_path / "session" / "future.json",
|
||||
_session("future", "2024/01/02 (Tue) 10:01", "future-secret"),
|
||||
)
|
||||
_write_jsonl(
|
||||
tmp_path / "first.jsonl",
|
||||
[
|
||||
{
|
||||
"question_id": "question-1",
|
||||
"answer": "reference one",
|
||||
"reason": "first reason",
|
||||
},
|
||||
],
|
||||
)
|
||||
_write_jsonl(
|
||||
tmp_path / "second.jsonl",
|
||||
[
|
||||
{
|
||||
"question_id": "question-1",
|
||||
"answer": "reference two",
|
||||
"reason": "second reason",
|
||||
},
|
||||
],
|
||||
)
|
||||
|
||||
wrapper = _FakeAgentWrapper(
|
||||
[
|
||||
'{"reason":"missing fence","golden_answer_correct":false,"answer":"invalid",'
|
||||
'"is_session_time_wrong":false}',
|
||||
'```json\n{"reason":"deprecated timestamp verdict","golden_answer_correct":false,'
|
||||
'"answer":"still invalid","is_session_time_wrong":true}\n```',
|
||||
"补充分析可以放在代码块外。\n"
|
||||
'```json\n{"reason":"由 past 和 equal 两个 session 支持 golden answer。",'
|
||||
'"golden_answer_correct":true,"answer":"","is_session_time_wrong":false}\n```\n'
|
||||
"审核完成。",
|
||||
],
|
||||
)
|
||||
sleep = AsyncMock()
|
||||
monkeypatch.setattr(review_module.asyncio, "sleep", sleep)
|
||||
app_context = ApplicationContext(
|
||||
workspace_dir=str(tmp_path),
|
||||
resource_dir="session",
|
||||
)
|
||||
step = FinalAnswerReviewStep(
|
||||
app_context=app_context,
|
||||
agent_wrapper=wrapper,
|
||||
reference_paths=["first.jsonl", "second.jsonl"],
|
||||
retry_initial_seconds=0.01,
|
||||
retry_max_seconds=0.02,
|
||||
)
|
||||
|
||||
response = asyncio.run(step())
|
||||
|
||||
assert response.success is True
|
||||
assert json.loads(response.answer) == {
|
||||
"reason": "由 past 和 equal 两个 session 支持 golden answer。",
|
||||
"golden_answer_correct": True,
|
||||
"answer": "",
|
||||
"is_session_time_wrong": False,
|
||||
}
|
||||
assert response.metadata["attempts"] == 3
|
||||
assert response.metadata["num_sessions"] == 3
|
||||
assert response.metadata["num_future_sessions"] == 1
|
||||
assert response.metadata["future_sessions"] == [
|
||||
{
|
||||
"session_id": "future",
|
||||
"session_date": "2024/01/02 (Tue) 10:01",
|
||||
"session_file": "future.json",
|
||||
},
|
||||
]
|
||||
assert len(wrapper.calls) == 3
|
||||
prompt, reply_kwargs = wrapper.calls[0]
|
||||
assert "past-evidence" not in prompt
|
||||
assert "equal-evidence" not in prompt
|
||||
assert "full-past-evidence" not in prompt
|
||||
assert "future-secret" not in prompt
|
||||
assert "extra_query_field" in prompt
|
||||
assert "extra_answer_field" in prompt
|
||||
assert "reference one" in prompt and "reference two" in prompt
|
||||
assert '"session_time_check"' in prompt
|
||||
assert '"sessions_after_question_date": [' in prompt
|
||||
assert '"answer_session_ids_after_question_date"' not in prompt
|
||||
assert '"future"' in prompt
|
||||
assert "output_schema" not in reply_kwargs
|
||||
assert [call.args for call in sleep.await_args_list] == [(0.01,), (0.02,)]
|
||||
|
||||
|
||||
# pylint: disable=protected-access
|
||||
def test_final_answer_review_reference_paths_env_overrides_config(tmp_path, monkeypatch):
|
||||
"""The batch driver can pass its selected reference files into the job process."""
|
||||
configured = tmp_path / "configured.jsonl"
|
||||
selected = tmp_path / "selected.jsonl"
|
||||
_write_jsonl(
|
||||
configured,
|
||||
[{"question_id": "question-1", "answer": "configured", "reason": "configured reason"}],
|
||||
)
|
||||
_write_jsonl(
|
||||
selected,
|
||||
[{"question_id": "question-1", "answer": "selected", "reason": "selected reason"}],
|
||||
)
|
||||
monkeypatch.setenv(REFERENCE_PATHS_ENV, json.dumps([str(selected)]))
|
||||
step = FinalAnswerReviewStep(
|
||||
app_context=ApplicationContext(workspace_dir=str(tmp_path)),
|
||||
reference_paths=[str(configured)],
|
||||
)
|
||||
|
||||
references = step._load_references("question-1")
|
||||
|
||||
assert len(references) == 1
|
||||
assert references[0]["answer"] == "selected"
|
||||
assert references[0]["source"] == selected.name
|
||||
|
||||
|
||||
def test_final_answer_review_allows_question_without_reference_answer(tmp_path):
|
||||
"""Samples outside the disputed lists are reviewed from answer.json alone."""
|
||||
references_path = tmp_path / "references.jsonl"
|
||||
_write_jsonl(
|
||||
references_path,
|
||||
[{"question_id": "another-question", "answer": "other", "reason": "other reason"}],
|
||||
)
|
||||
step = FinalAnswerReviewStep(
|
||||
app_context=ApplicationContext(workspace_dir=str(tmp_path)),
|
||||
reference_paths=[str(references_path)],
|
||||
)
|
||||
|
||||
assert not step._load_references("question-without-reference")
|
||||
|
||||
|
||||
# pylint: enable=protected-access
|
||||
|
||||
|
||||
def test_final_answer_review_agent_cwd_is_sample_session_directory(tmp_path):
|
||||
"""The configured relative cwd resolves inside each selected LME workspace."""
|
||||
config = resolve_app_config(config="jinli_lme", log_config=False)
|
||||
agent_config = config["components"]["agent_wrapper"]["lme_final_answer_review"]
|
||||
assert agent_config["cwd"] == "session"
|
||||
|
||||
wrapper = CcAgentWrapper(
|
||||
app_context=ApplicationContext(workspace_dir=str(tmp_path)),
|
||||
cwd=agent_config["cwd"],
|
||||
)
|
||||
assert wrapper.cwd == tmp_path / "session"
|
||||
|
||||
|
||||
# pylint: disable=protected-access
|
||||
def test_final_answer_review_requires_empty_answer_when_golden_is_correct():
|
||||
"""Correct golden answers are collected without duplicating their answer text."""
|
||||
parsed = FinalAnswerReviewStep._parse_reply(
|
||||
'```json\n{"reason":"golden is supported","golden_answer_correct":true,"answer":"",'
|
||||
'"is_session_time_wrong":false}\n```',
|
||||
)
|
||||
assert parsed == {
|
||||
"reason": "golden is supported",
|
||||
"golden_answer_correct": True,
|
||||
"answer": "",
|
||||
"is_session_time_wrong": False,
|
||||
}
|
||||
|
||||
with pytest.raises(ValueError, match="answer.*must be empty"):
|
||||
FinalAnswerReviewStep._parse_reply(
|
||||
'```json\n{"reason":"bad duplicate","golden_answer_correct":true,"answer":"duplicate",'
|
||||
'"is_session_time_wrong":false}\n```',
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError, match="deprecated and must be false"):
|
||||
FinalAnswerReviewStep._parse_reply(
|
||||
'```json\n{"reason":"legacy session id verdict",'
|
||||
'"golden_answer_correct":false,"answer":"corrected","is_session_time_wrong":true}\n```',
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError, match="must not evaluate answer_session_ids"):
|
||||
FinalAnswerReviewStep._parse_reply(
|
||||
'```json\n{"reason":"answer_session_ids contains a future session",'
|
||||
'"golden_answer_correct":false,"answer":"corrected","is_session_time_wrong":false}\n```',
|
||||
)
|
||||
|
||||
|
||||
# pylint: enable=protected-access
|
||||
|
||||
|
||||
def test_final_answer_review_rejects_unparseable_session_time_before_agent(tmp_path):
|
||||
"""An unknown session time is never silently admitted across the time boundary."""
|
||||
_write_json(
|
||||
tmp_path / "query.json",
|
||||
{
|
||||
"question_id": "question-1",
|
||||
"question": "Q",
|
||||
"question_date": "2024/01/02 (Tue) 10:00",
|
||||
},
|
||||
)
|
||||
_write_json(tmp_path / "answer.json", {"answer": "A"})
|
||||
_write_json(
|
||||
tmp_path / "session" / "bad.json",
|
||||
_session("bad", "unknown", "must-not-reach-agent"),
|
||||
)
|
||||
_write_jsonl(
|
||||
tmp_path / "refs.jsonl",
|
||||
[{"question_id": "question-1", "answer": "reference", "reason": "reason"}],
|
||||
)
|
||||
valid_reply = "".join(
|
||||
[
|
||||
'```json\n{"reason":"y","golden_answer_correct":false,',
|
||||
'"answer":"x","is_session_time_wrong":false}\n```',
|
||||
],
|
||||
)
|
||||
wrapper = _FakeAgentWrapper([valid_reply])
|
||||
step = FinalAnswerReviewStep(
|
||||
app_context=ApplicationContext(
|
||||
workspace_dir=str(tmp_path),
|
||||
resource_dir="session",
|
||||
),
|
||||
agent_wrapper=wrapper,
|
||||
reference_paths=["refs.jsonl"],
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError, match="Invalid LongMemEval datetime"):
|
||||
asyncio.run(step())
|
||||
assert not wrapper.calls
|
||||
|
||||
|
||||
def test_driver_merges_references_and_atomically_rewrites_in_input_order(tmp_path):
|
||||
"""The batch checkpoint contains one stable row per completed question."""
|
||||
first = tmp_path / "first.jsonl"
|
||||
second = tmp_path / "second.jsonl"
|
||||
_write_jsonl(
|
||||
first,
|
||||
[
|
||||
{"question_id": "q2", "answer": "a2", "reason": "r2"},
|
||||
{"question_id": "q1", "answer": "a1", "reason": "r1"},
|
||||
],
|
||||
)
|
||||
_write_jsonl(second, [{"question_id": "q1", "answer": "a1b", "reason": "r1b"}])
|
||||
|
||||
merged = merge_references([first, second])
|
||||
|
||||
assert list(merged) == ["q2", "q1"]
|
||||
assert len(merged["q2"]) == 1
|
||||
assert len(merged["q1"]) == 2
|
||||
output = tmp_path / "result.jsonl"
|
||||
atomic_write_results(
|
||||
output,
|
||||
list(merged),
|
||||
{
|
||||
"q1": {
|
||||
"reason": "reason-1",
|
||||
"golden_answer_correct": False,
|
||||
"answer": "final-1",
|
||||
"is_session_time_wrong": False,
|
||||
},
|
||||
"q2": {
|
||||
"reason": "reason-2",
|
||||
"golden_answer_correct": False,
|
||||
"answer": "final-2",
|
||||
"is_session_time_wrong": True,
|
||||
},
|
||||
},
|
||||
)
|
||||
rows = _read_output(output)
|
||||
assert [row["question_id"] for row in rows] == ["q2", "q1"]
|
||||
assert driver_module.load_existing(output)["q2"]["is_session_time_wrong"] is False
|
||||
|
||||
|
||||
def test_driver_selects_all_or_explicit_question_ids(tmp_path):
|
||||
"""Explicit IDs may select samples that have no reference-answer row."""
|
||||
mapping = {
|
||||
"q1": tmp_path / "0",
|
||||
"q2": tmp_path / "1",
|
||||
"q3": tmp_path / "2",
|
||||
}
|
||||
|
||||
assert select_question_ids(mapping, None) == ["q1", "q2", "q3"]
|
||||
assert select_question_ids(mapping, ["q3", "q1"]) == ["q3", "q1"]
|
||||
assert select_question_ids(mapping, None, {"q1", "q3"}) == ["q2"]
|
||||
assert select_question_ids(mapping, ["q3", "q2"], {"q3"}) == ["q2"]
|
||||
with pytest.raises(ValueError, match="No dataset workspace"):
|
||||
select_question_ids(mapping, ["unknown"])
|
||||
with pytest.raises(ValueError, match="Duplicate"):
|
||||
select_question_ids(mapping, ["q1", "q1"])
|
||||
|
||||
|
||||
def test_driver_limits_concurrency_and_spaces_submissions(tmp_path, monkeypatch):
|
||||
"""Concurrent jobs never exceed the cap and are not submitted in a burst."""
|
||||
mapping = {f"q{index}": tmp_path / str(index) for index in range(4)}
|
||||
starts: list[float] = []
|
||||
active = 0
|
||||
max_active = 0
|
||||
lock = threading.Lock()
|
||||
|
||||
def fake_run_one(question_id, workspace, log_dir, reference_paths):
|
||||
del question_id, workspace, log_dir, reference_paths
|
||||
nonlocal active, max_active
|
||||
with lock:
|
||||
starts.append(time.monotonic())
|
||||
active += 1
|
||||
max_active = max(max_active, active)
|
||||
time.sleep(0.055)
|
||||
with lock:
|
||||
active -= 1
|
||||
return {
|
||||
"reason": "reviewed",
|
||||
"golden_answer_correct": True,
|
||||
"answer": "",
|
||||
"is_session_time_wrong": False,
|
||||
}
|
||||
|
||||
monkeypatch.setattr(driver_module, "workspace_map", lambda: mapping)
|
||||
monkeypatch.setattr(driver_module, "merge_references", lambda paths: {})
|
||||
monkeypatch.setattr(driver_module, "load_existing", lambda path: {})
|
||||
monkeypatch.setattr(driver_module, "atomic_write_results", lambda *args: None)
|
||||
monkeypatch.setattr(driver_module, "run_one", fake_run_one)
|
||||
monkeypatch.setattr(driver_module, "MIN_SUBMIT_INTERVAL_SECONDS", 0.0)
|
||||
monkeypatch.setattr(
|
||||
sys,
|
||||
"argv",
|
||||
[
|
||||
"run_final_answer_review.py",
|
||||
"--concurrency",
|
||||
"3",
|
||||
"--submit-interval-seconds",
|
||||
"0.02",
|
||||
"--output",
|
||||
str(tmp_path / "output.jsonl"),
|
||||
],
|
||||
)
|
||||
|
||||
assert driver_module.main() == 0
|
||||
assert max_active == 3
|
||||
assert len(starts) == 4
|
||||
assert all(later - earlier >= 0.015 for earlier, later in zip(starts, starts[1:]))
|
||||
|
||||
|
||||
def _read_output(path: Path) -> list[dict]:
|
||||
return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines()]
|
||||
Loading…
Add table
Reference in a new issue