From 51eb53e831f18b68cac54206b8b8f4d7f22cf243 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=96=B9=E5=BA=94?= Date: Fri, 13 Mar 2026 14:22:10 +0800 Subject: [PATCH] =?UTF-8?q?docs(readme):=20=E6=9B=B4=E6=96=B0=E6=96=87?= =?UTF-8?q?=E6=A1=A3=E6=B7=BB=E5=8A=A0=E5=AE=9E=E9=AA=8C=E7=BB=93=E6=9E=9C?= =?UTF-8?q?=E8=A1=A8=E6=A0=BC?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - 在英文版 README 中添加 🧪 Experiments 章节 - 添加 LoCoMo 和 HaluMem 两个基准测试的结果表格 - 在中文版 README_ZH 中添加 🧪 实验 章节 - 添加 LoCoMo 和 HaluMem 测试集的实验配置说明 - 添加完整的实验数据对比表格和评估协议说明 --- README.md | 38 ++++++++++++++++++++++++++++++++++++++ README_ZH.md | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 76 insertions(+) diff --git a/README.md b/README.md index ca7726bd..1e0f2740 100644 --- a/README.md +++ b/README.md @@ -416,6 +416,44 @@ memories: Installation and environment configuration are the same as [ReMeLight](#installation). API keys are configured via environment variables and can be stored in a `.env` file at the project root. +## 🧪 Experiments + +Evaluations are conducted on three benchmarks: **LoCoMo** and **HaluMem**. Experimental settings: + +1. **ReMe backbone**: as specified in each table. +2. **Evaluation protocol**: LLM-as-a-Judge following MemOS — each answer is scored by GPT-4o-mini. + +Baseline results are reproduced from their respective papers under aligned settings where possible. + + + +### LoCoMo + +| Method | Single Hop | Multi Hop | Temporal | Open Domain | Overall | +|--------|------------|-----------|-----------|-------------|-----------| +| MemoryOS | 62.43 | 56.50 | 37.18 | 40.28 | 54.70 | +| Mem0 | 66.71 | 58.16 | 55.45 | 40.62 | 61.00 | +| MemU | 72.77 | 62.41 | 33.96 | 46.88 | 61.15 | +| MemOS | 81.45 | 69.15 | 72.27 | 60.42 | 75.87 | +| HiMem | 89.22 | 70.92 | 74.77 | 54.86 | 80.71 | +| Zep | 88.11 | 71.99 | 74.45 | 66.67 | 81.06 | +| TiMem | 81.43 | 62.20 | 77.63 | 52.08 | 75.30 | +| TSM | 84.30 | 66.67 | 71.03 | 58.33 | 76.69 | +| MemR3 | 89.44 | 71.39 | 76.22 | 61.11 | 81.55 | +| **ReMe** | **89.89** | **82.98** | **83.80** | **71.88** | **86.23** | + + +### HaluMem + +| Method | Memory Integrity | Memory Accuracy | QA Accuracy | +|-------------|------------------|---------------|-------------| +| MemoBase | 14.55 | 92.24 | 35.53 | +| Supermemory | 41.53 | 90.32 | 54.07 | +| Mem0 | 42.91 | 86.26 | 53.02 | +| ProMem | **73.80** | 89.47 | 62.26 | +| **ReMe** | 67.72 | **94.06** | **88.78** | + + ### Python usage ```python diff --git a/README_ZH.md b/README_ZH.md index c3de853f..6081ab37 100644 --- a/README_ZH.md +++ b/README_ZH.md @@ -396,6 +396,44 @@ graph LR 安装和环境变量配置与 [ReMeLight 一致](#安装)，通过环境变量设置 API 密钥，可写在项目根目录的 `.env` 文件中。 +## 🧪 实验 + +本实验部分在 LoCoMo、LongMemEval、HaluMem 三个数据集上进行评测，实验设置如下： + +1. **ReMe 使用模型**：如各表 backbone 列所示。 +2. **评估使用模型**：采用 LLM-as-a-Judge 协议（参照 MemOS）——每条回答由 GPT-4o-mini 裁判模型打分。 + +实验设置尽量与各基线论文保持一致，以复用其公开结果。 + + +### LoCoMo + +| Method | Single Hop | Multi Hop | Temporal | Open Domain | Overall | +|--------|------------|-----------|-----------|-------------|-----------| +| MemoryOS | 62.43 | 56.50 | 37.18 | 40.28 | 54.70 | +| Mem0 | 66.71 | 58.16 | 55.45 | 40.62 | 61.00 | +| MemU | 72.77 | 62.41 | 33.96 | 46.88 | 61.15 | +| MemOS | 81.45 | 69.15 | 72.27 | 60.42 | 75.87 | +| HiMem | 89.22 | 70.92 | 74.77 | 54.86 | 80.71 | +| Zep | 88.11 | 71.99 | 74.45 | 66.67 | 81.06 | +| TiMem | 81.43 | 62.20 | 77.63 | 52.08 | 75.30 | +| TSM | 84.30 | 66.67 | 71.03 | 58.33 | 76.69 | +| MemR3 | 89.44 | 71.39 | 76.22 | 61.11 | 81.55 | +| **ReMe** | **89.89** | **82.98** | **83.80** | **71.88** | **86.23** | + + +### HaluMem + +| Method | Memory Integrity | Memory Accuracy | QA Accuracy | +|-------------|------------------|---------------|-------------| +| MemoBase | 14.55 | 92.24 | 35.53 | +| Supermemory | 41.53 | 90.32 | 54.07 | +| Mem0 | 42.91 | 86.26 | 53.02 | +| ProMem | **73.80** | 89.47 | 62.26 | +| **ReMe** | 67.72 | **94.06** | **88.78** | + + + ### Python 使用 ```python