From 935e886af37ef357a844d07152dd4df26710d0eb Mon Sep 17 00:00:00 2001 From: Zhouwk <57825291+nitwtog@users.noreply.github.com> Date: Tue, 7 Apr 2026 14:20:30 +0800 Subject: [PATCH] =?UTF-8?q?=E6=9B=B4=E6=96=B0longmemeval=E5=92=8Chalumem?= =?UTF-8?q?=E7=9A=84quick=20start=20(#194)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(reme): 添加配置选项以启用或禁用个人资料功能 - 在 ReMe 初始化方法中添加 enable_profile 参数，默认值为 True - 根据 enable_profile 设置决定是否创建 profile 目录和设置 profile_dir - 在 PersonalSummarizer 中根据 enable_profile 条件性地添加个人资料相关工具 - 在 PersonalRetriever 中根据 enable_profile 条件性地添加 ReadAllProfiles 工具 - 修改 profile_path 属性以在禁用个人资料时返回 None - 修改 get_profile_handler 方法以在禁用个人资料时返回 None - 为 enable_profile 参数添加文档说明其用于云向量存储场景 * refactor(benchmark): 重构LongMemEval基准测试中的ReMe实例管理 - 移除未使用的shutil导入 - 将固定的ReMe实例改为每个问题创建独立实例以实现隔离 - 更新LLM配置名称从qwen3-max-think到qwen-max-t - 修改模型调用逻辑使用正确的model_name参数 - 添加qwen-flash和GPT-4o-mini等新模型配置 - 统一使用"User"作为用户名，通过集合名实现隔离 - 调整并发处理数从4降至1，批处理大小从10增至30 - 每个问题类型采样数从2增至4 - 添加异步上下文管理确保资源正确释放 * reformat 2 files * refactor(benchmark): 重构长记忆评估中的模型配置 - 将原有的 eval_model_name 替换为专门的 retrieve_model_name 用于检索操作 - 添加对 qwen-max 模型配置的支持 - 更新参数解析器以支持新的检索模型参数 - 修改最大并发数默认值从 1 提升到 4 - 调整样本数量默认值从 4 减少到 1 - 统一模型参数命名规范，区分摘要、检索和评估模型 - 优化内存处理器初始化逻辑，支持独立的检索模型配置 * fix(benchmark): 移除数据路径默认值并设为必填参数 - 将LongMemEval评估脚本中的data_path参数改为必需参数 - 将HaluMem评估脚本中的data_path参数改为必需参数 - 删除了硬编码的默认文件路径配置 - 强制用户显式指定数据集文件路径以避免路径错误 * Update __init__.py * Update __init__.py * fix(benchmark): 修复ReMe评估中的模型配置和空值处理问题 - 移除了retrieve_memory调用中不需要的llm_config_name参数 - 修复了长字符串打印的换行格式问题 - 添加了eval_result为空时的初始化处理 - 在accuracy评估中加入了eval_model_name参数传递 * style(benchmark): 格式化模型名称打印输出 - 移除了多行字符串中的换行符和多余空格 - 将模型名称信息合并为单行连续显示 - 保持了原有的打印格式和信息完整性 * docs(readme): 更新文档添加实验结果表格 - 在英文版 README 中添加 🧪 Experiments 章节 - 添加 LoCoMo 和 HaluMem 两个基准测试的结果表格 - 在中文版 README_ZH 中添加 🧪 实验 章节 - 添加 LoCoMo 和 HaluMem 测试集的实验配置说明 - 添加完整的实验数据对比表格和评估协议说明 * docs(readme): 更新文档中的内存系统链接 - 为基于文件的记忆系统添加锚点链接 - 为基于向量库的记忆系统添加锚点链接 - 修复英文文档中的链接格式 - 修复中文文档中的链接格式和空行问题 * docs(readme): update experimental results section in documentation - Remove outdated experimental data placeholder "Coming soon..." - Add complete evaluation results for LoCoMo and HaluMem benchmarks - Include detailed performance metrics tables for all memory methods - Update experimental settings description with ReMe backbone details - Align evaluation protocol information with LLM-as-a-Judge approach - Maintain consistent formatting between English and Chinese documentation * docs(benchmark): add quick start guides for halumem and longmemeval experiments - Created HaluMem experiment quick start guide with ReMe integration setup - Added detailed steps for installing ReMe environment using conda - Included repository cloning instructions for HaluMem benchmark - Provided complete command examples for running HaluMem experiments - Created LongMeMEval quick start guide with data download procedures - Added wget commands for downloading cleaned dataset files - Included evaluation script instructions for computing experiment statistics - Documented parameter configurations for different model types and batch sizes * docs(longmemeval): update quickstart guide documentation - Changed project name from Halumem to Longmemeval in title - Updated description to reference Longmemeval experiments instead of Halumem - Maintained existing ReMe integration instructions unchanged --- benchmark/halumem/quickstart.md | 33 ++++++++++++++++++++ benchmark/longmemeval/quickstart.md | 48 +++++++++++++++++++++++++++++ 2 files changed, 81 insertions(+) create mode 100644 benchmark/halumem/quickstart.md create mode 100644 benchmark/longmemeval/quickstart.md diff --git a/benchmark/halumem/quickstart.md b/benchmark/halumem/quickstart.md new file mode 100644 index 00000000..59dc68fa --- /dev/null +++ b/benchmark/halumem/quickstart.md @@ -0,0 +1,33 @@ +# Halumem +Experiment Quick Start Guide +This guide helps you quickly set up and run Halumem experiments with ReMe integration. + +### 1. Start ReMe Service +Install ReMe (if not already installed) +If you haven't installed the ReMe environment yet, follow these steps: +```bash +# Create ReMe environment +conda create -p ./reme-env python==3.12 +conda activate ./reme-env + +# Install ReMe +pip install . +``` + +### 2. Clone the Repository +```bash +cd ./benchmark/halumem +git clone https://github.com/MemTensor/HaluMem.git +``` + +### 3. Run Experiments +Launch the ReMe service to enable memory library functionality: +```bash +clear && python benchmark/halumem/eval_reme.py \ + --data_path benchmark/halumem/HaluMem/data/HaluMem-Medium.jsonl \ + --reme_model_name gpt-4o-mini-2024-07-18 \ + --eval_model_name gpt-4o-mini-2024-07-18 \ + --batch_size 40 \ + --algo_version default +``` + diff --git a/benchmark/longmemeval/quickstart.md b/benchmark/longmemeval/quickstart.md new file mode 100644 index 00000000..c27d5c65 --- /dev/null +++ b/benchmark/longmemeval/quickstart.md @@ -0,0 +1,48 @@ +# Longmemeval +Experiment Quick Start Guide +This guide helps you quickly set up and run Longmemeval experiments with ReMe integration. + +### 1. Start ReMe Service +Install ReMe (if not already installed) +If you haven't installed the ReMe environment yet, follow these steps: +```bash +# Create ReMe environment +conda create -p ./reme-env python==3.12 +conda activate ./reme-env + +# Install ReMe +pip install . +``` + +### 2. Clone the Repository +```bash +cd ./benchmark/longmemeval +mkdir -p data/ +cd data/ +wget https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_oracle.json +wget https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json +wget https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_m_cleaned.json +cd .. +``` + +### 3. Run Experiments +Launch the ReMe service to enable memory library functionality: +```bash +clear && python benchmark/longmemeval/eval_longmemeval_reme.py \ + --data_path benchmark/longmemeval/data/longmemeval_s_cleaned.json \ + --reme_model_name qwen-flash \ + --reme_model_name retrieve_model_name \ + --eval_model_name gpt-4o-mini-2024-07-18 \ + --batch_size 20 \ + --algo_version default +``` + + +### 4. Evaluate Results +Evaluate the results of the experiments: +```bash +python benchmark/longmememeval/compute_stats.py \ + --results_dir bench_results/longmemeval_reme \ + --output_file bench_results/longmemeval_reme/statistics.json +``` +The `compute_stats.py` script computes various statistics from the evaluation results. \ No newline at end of file