diff --git a/README.md b/README.md index f379652f..05a7c29d 100644 --- a/README.md +++ b/README.md @@ -415,43 +415,6 @@ memories: Installation and environment configuration are the same as [ReMeLight](#installation). API keys are configured via environment variables and can be stored in a `.env` file at the project root. -## 🧪 Experiments - -Evaluations are conducted on three benchmarks: **LoCoMo** and **HaluMem**. Experimental settings: - -1. **ReMe backbone**: as specified in each table. -2. **Evaluation protocol**: LLM-as-a-Judge following MemOS — each answer is scored by GPT-4o-mini. - -Baseline results are reproduced from their respective papers under aligned settings where possible. - - - -### LoCoMo - -| Method | Single Hop | Multi Hop | Temporal | Open Domain | Overall | -|--------|------------|-----------|-----------|-------------|-----------| -| MemoryOS | 62.43 | 56.50 | 37.18 | 40.28 | 54.70 | -| Mem0 | 66.71 | 58.16 | 55.45 | 40.62 | 61.00 | -| MemU | 72.77 | 62.41 | 33.96 | 46.88 | 61.15 | -| MemOS | 81.45 | 69.15 | 72.27 | 60.42 | 75.87 | -| HiMem | 89.22 | 70.92 | 74.77 | 54.86 | 80.71 | -| Zep | 88.11 | 71.99 | 74.45 | 66.67 | 81.06 | -| TiMem | 81.43 | 62.20 | 77.63 | 52.08 | 75.30 | -| TSM | 84.30 | 66.67 | 71.03 | 58.33 | 76.69 | -| MemR3 | 89.44 | 71.39 | 76.22 | 61.11 | 81.55 | -| **ReMe** | **89.89** | **82.98** | **83.80** | **71.88** | **86.23** | - - -### HaluMem - -| Method | Memory Integrity | Memory Accuracy | QA Accuracy | -|-------------|------------------|---------------|-------------| -| MemoBase | 14.55 | 92.24 | 35.53 | -| Supermemory | 41.53 | 90.32 | 54.07 | -| Mem0 | 42.91 | 86.26 | 53.02 | -| ProMem | **73.80** | 89.47 | 62.26 | -| **ReMe** | 67.72 | **94.06** | **88.78** | - ### Python usage @@ -568,7 +531,38 @@ graph LR ### Experimental results -Coming soon... +Evaluations are conducted on Two benchmarks: **LoCoMo** and **HaluMem**. Experimental settings: + +1. **ReMe backbone**: as specified in each table. +2. **Evaluation protocol**: LLM-as-a-Judge following MemOS — each answer is scored by GPT-4o-mini. + +Baseline results are reproduced from their respective papers under aligned settings where possible. + +### LoCoMo + +| Method | Single Hop | Multi Hop | Temporal | Open Domain | Overall | +|--------|------------|-----------|-----------|-------------|-----------| +| MemoryOS | 62.43 | 56.50 | 37.18 | 40.28 | 54.70 | +| Mem0 | 66.71 | 58.16 | 55.45 | 40.62 | 61.00 | +| MemU | 72.77 | 62.41 | 33.96 | 46.88 | 61.15 | +| MemOS | 81.45 | 69.15 | 72.27 | 60.42 | 75.87 | +| HiMem | 89.22 | 70.92 | 74.77 | 54.86 | 80.71 | +| Zep | 88.11 | 71.99 | 74.45 | 66.67 | 81.06 | +| TiMem | 81.43 | 62.20 | 77.63 | 52.08 | 75.30 | +| TSM | 84.30 | 66.67 | 71.03 | 58.33 | 76.69 | +| MemR3 | 89.44 | 71.39 | 76.22 | 61.11 | 81.55 | +| **ReMe** | **89.89** | **82.98** | **83.80** | **71.88** | **86.23** | + + +### HaluMem + +| Method | Memory Integrity | Memory Accuracy | QA Accuracy | +|-------------|------------------|---------------|-------------| +| MemoBase | 14.55 | 92.24 | 35.53 | +| Supermemory | 41.53 | 90.32 | 54.07 | +| Mem0 | 42.91 | 86.26 | 53.02 | +| ProMem | **73.80** | 89.47 | 62.26 | +| **ReMe** | 67.72 | **94.06** | **88.78** | --- diff --git a/README_ZH.md b/README_ZH.md index 3d37e30f..45c1b277 100644 --- a/README_ZH.md +++ b/README_ZH.md @@ -397,43 +397,6 @@ graph LR 安装和环境变量配置与 [ReMeLight 一致](#安装)，通过环境变量设置 API 密钥，可写在项目根目录的 `.env` 文件中。 -## 🧪 实验 - -本实验部分在 LoCoMo、LongMemEval、HaluMem 三个数据集上进行评测，实验设置如下： - -1. **ReMe 使用模型**：如各表 backbone 列所示。 -2. **评估使用模型**：采用 LLM-as-a-Judge 协议（参照 MemOS）——每条回答由 GPT-4o-mini 裁判模型打分。 - -实验设置尽量与各基线论文保持一致，以复用其公开结果。 - - -### LoCoMo - -| Method | Single Hop | Multi Hop | Temporal | Open Domain | Overall | -|--------|------------|-----------|-----------|-------------|-----------| -| MemoryOS | 62.43 | 56.50 | 37.18 | 40.28 | 54.70 | -| Mem0 | 66.71 | 58.16 | 55.45 | 40.62 | 61.00 | -| MemU | 72.77 | 62.41 | 33.96 | 46.88 | 61.15 | -| MemOS | 81.45 | 69.15 | 72.27 | 60.42 | 75.87 | -| HiMem | 89.22 | 70.92 | 74.77 | 54.86 | 80.71 | -| Zep | 88.11 | 71.99 | 74.45 | 66.67 | 81.06 | -| TiMem | 81.43 | 62.20 | 77.63 | 52.08 | 75.30 | -| TSM | 84.30 | 66.67 | 71.03 | 58.33 | 76.69 | -| MemR3 | 89.44 | 71.39 | 76.22 | 61.11 | 81.55 | -| **ReMe** | **89.89** | **82.98** | **83.80** | **71.88** | **86.23** | - - -### HaluMem - -| Method | Memory Integrity | Memory Accuracy | QA Accuracy | -|-------------|------------------|---------------|-------------| -| MemoBase | 14.55 | 92.24 | 35.53 | -| Supermemory | 41.53 | 90.32 | 54.07 | -| Mem0 | 42.91 | 86.26 | 53.02 | -| ProMem | **73.80** | 89.47 | 62.26 | -| **ReMe** | 67.72 | **94.06** | **88.78** | - - ### Python 使用 @@ -550,8 +513,39 @@ graph LR ### 实验效果 -Coming soon... +本实验部分在 LoCoMo、HaluMem 两个数据集上进行评测，实验设置如下： +1. **ReMe 使用模型**：如各表 backbone 列所示。 +2. **评估使用模型**：采用 LLM-as-a-Judge 协议（参照 MemOS）——每条回答由 GPT-4o-mini 裁判模型打分。 + +实验设置尽量与各基线论文保持一致，以复用其公开结果。 + + +#### LoCoMo + +| Method | Single Hop | Multi Hop | Temporal | Open Domain | Overall | +|--------|------------|-----------|-----------|-------------|-----------| +| MemoryOS | 62.43 | 56.50 | 37.18 | 40.28 | 54.70 | +| Mem0 | 66.71 | 58.16 | 55.45 | 40.62 | 61.00 | +| MemU | 72.77 | 62.41 | 33.96 | 46.88 | 61.15 | +| MemOS | 81.45 | 69.15 | 72.27 | 60.42 | 75.87 | +| HiMem | 89.22 | 70.92 | 74.77 | 54.86 | 80.71 | +| Zep | 88.11 | 71.99 | 74.45 | 66.67 | 81.06 | +| TiMem | 81.43 | 62.20 | 77.63 | 52.08 | 75.30 | +| TSM | 84.30 | 66.67 | 71.03 | 58.33 | 76.69 | +| MemR3 | 89.44 | 71.39 | 76.22 | 61.11 | 81.55 | +| **ReMe** | **89.89** | **82.98** | **83.80** | **71.88** | **86.23** | + + +#### HaluMem + +| Method | Memory Integrity | Memory Accuracy | QA Accuracy | +|-------------|------------------|---------------|-------------| +| MemoBase | 14.55 | 92.24 | 35.53 | +| Supermemory | 41.53 | 90.32 | 54.07 | +| Mem0 | 42.91 | 86.26 | 53.02 | +| ProMem | **73.80** | 89.47 | 62.26 | +| **ReMe** | 67.72 | **94.06** | **88.78** | --- ## 🧪 程序化记忆论文