mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-09-14 23:21:04 +00:00
docs(readme): update experimental results section in documentation
- Remove outdated experimental data placeholder "Coming soon..." - Add complete evaluation results for LoCoMo and HaluMem benchmarks - Include detailed performance metrics tables for all memory methods - Update experimental settings description with ReMe backbone details - Align evaluation protocol information with LLM-as-a-Judge approach - Maintain consistent formatting between English and Chinese documentation
This commit is contained in:
方应
parent
eb2cdb3aef
commit
839c30d022
2 changed files with 64 additions and 76 deletions
70
README.md
70
README.md
|
|
@ -415,43 +415,6 @@ memories:
|
|||
Installation and environment configuration are the same as [ReMeLight](#installation).
|
||||
API keys are configured via environment variables and can be stored in a `.env` file at the project root.
|
||||
|
||||
## 🧪 Experiments
|
||||
|
||||
Evaluations are conducted on three benchmarks: **LoCoMo** and **HaluMem**. Experimental settings:
|
||||
|
||||
1. **ReMe backbone**: as specified in each table.
|
||||
2. **Evaluation protocol**: LLM-as-a-Judge following MemOS — each answer is scored by GPT-4o-mini.
|
||||
|
||||
Baseline results are reproduced from their respective papers under aligned settings where possible.
|
||||
|
||||
|
||||
|
||||
### LoCoMo
|
||||
|
||||
| Method | Single Hop | Multi Hop | Temporal | Open Domain | Overall |
|
||||
|--------|------------|-----------|-----------|-------------|-----------|
|
||||
| MemoryOS | 62.43 | 56.50 | 37.18 | 40.28 | 54.70 |
|
||||
| Mem0 | 66.71 | 58.16 | 55.45 | 40.62 | 61.00 |
|
||||
| MemU | 72.77 | 62.41 | 33.96 | 46.88 | 61.15 |
|
||||
| MemOS | 81.45 | 69.15 | 72.27 | 60.42 | 75.87 |
|
||||
| HiMem | 89.22 | 70.92 | 74.77 | 54.86 | 80.71 |
|
||||
| Zep | 88.11 | 71.99 | 74.45 | 66.67 | 81.06 |
|
||||
| TiMem | 81.43 | 62.20 | 77.63 | 52.08 | 75.30 |
|
||||
| TSM | 84.30 | 66.67 | 71.03 | 58.33 | 76.69 |
|
||||
| MemR3 | 89.44 | 71.39 | 76.22 | 61.11 | 81.55 |
|
||||
| **ReMe** | **89.89** | **82.98** | **83.80** | **71.88** | **86.23** |
|
||||
|
||||
|
||||
### HaluMem
|
||||
|
||||
| Method | Memory Integrity | Memory Accuracy | QA Accuracy |
|
||||
|-------------|------------------|---------------|-------------|
|
||||
| MemoBase | 14.55 | 92.24 | 35.53 |
|
||||
| Supermemory | 41.53 | 90.32 | 54.07 |
|
||||
| Mem0 | 42.91 | 86.26 | 53.02 |
|
||||
| ProMem | **73.80** | 89.47 | 62.26 |
|
||||
| **ReMe** | 67.72 | **94.06** | **88.78** |
|
||||
|
||||
|
||||
### Python usage
|
||||
|
||||
|
|
@ -568,7 +531,38 @@ graph LR
|
|||
|
||||
### Experimental results
|
||||
|
||||
Coming soon...
|
||||
Evaluations are conducted on Two benchmarks: **LoCoMo** and **HaluMem**. Experimental settings:
|
||||
|
||||
1. **ReMe backbone**: as specified in each table.
|
||||
2. **Evaluation protocol**: LLM-as-a-Judge following MemOS — each answer is scored by GPT-4o-mini.
|
||||
|
||||
Baseline results are reproduced from their respective papers under aligned settings where possible.
|
||||
|
||||
### LoCoMo
|
||||
|
||||
| Method | Single Hop | Multi Hop | Temporal | Open Domain | Overall |
|
||||
|--------|------------|-----------|-----------|-------------|-----------|
|
||||
| MemoryOS | 62.43 | 56.50 | 37.18 | 40.28 | 54.70 |
|
||||
| Mem0 | 66.71 | 58.16 | 55.45 | 40.62 | 61.00 |
|
||||
| MemU | 72.77 | 62.41 | 33.96 | 46.88 | 61.15 |
|
||||
| MemOS | 81.45 | 69.15 | 72.27 | 60.42 | 75.87 |
|
||||
| HiMem | 89.22 | 70.92 | 74.77 | 54.86 | 80.71 |
|
||||
| Zep | 88.11 | 71.99 | 74.45 | 66.67 | 81.06 |
|
||||
| TiMem | 81.43 | 62.20 | 77.63 | 52.08 | 75.30 |
|
||||
| TSM | 84.30 | 66.67 | 71.03 | 58.33 | 76.69 |
|
||||
| MemR3 | 89.44 | 71.39 | 76.22 | 61.11 | 81.55 |
|
||||
| **ReMe** | **89.89** | **82.98** | **83.80** | **71.88** | **86.23** |
|
||||
|
||||
|
||||
### HaluMem
|
||||
|
||||
| Method | Memory Integrity | Memory Accuracy | QA Accuracy |
|
||||
|-------------|------------------|---------------|-------------|
|
||||
| MemoBase | 14.55 | 92.24 | 35.53 |
|
||||
| Supermemory | 41.53 | 90.32 | 54.07 |
|
||||
| Mem0 | 42.91 | 86.26 | 53.02 |
|
||||
| ProMem | **73.80** | 89.47 | 62.26 |
|
||||
| **ReMe** | 67.72 | **94.06** | **88.78** |
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
70
README_ZH.md
70
README_ZH.md
|
|
@ -397,43 +397,6 @@ graph LR
|
|||
|
||||
安装和环境变量配置与 [ReMeLight 一致](#安装)，通过环境变量设置 API 密钥，可写在项目根目录的 `.env` 文件中。
|
||||
|
||||
## 🧪 实验
|
||||
|
||||
本实验部分在 LoCoMo、LongMemEval、HaluMem 三个数据集上进行评测，实验设置如下：
|
||||
|
||||
1. **ReMe 使用模型**：如各表 backbone 列所示。
|
||||
2. **评估使用模型**：采用 LLM-as-a-Judge 协议（参照 MemOS）——每条回答由 GPT-4o-mini 裁判模型打分。
|
||||
|
||||
实验设置尽量与各基线论文保持一致，以复用其公开结果。
|
||||
|
||||
|
||||
### LoCoMo
|
||||
|
||||
| Method | Single Hop | Multi Hop | Temporal | Open Domain | Overall |
|
||||
|--------|------------|-----------|-----------|-------------|-----------|
|
||||
| MemoryOS | 62.43 | 56.50 | 37.18 | 40.28 | 54.70 |
|
||||
| Mem0 | 66.71 | 58.16 | 55.45 | 40.62 | 61.00 |
|
||||
| MemU | 72.77 | 62.41 | 33.96 | 46.88 | 61.15 |
|
||||
| MemOS | 81.45 | 69.15 | 72.27 | 60.42 | 75.87 |
|
||||
| HiMem | 89.22 | 70.92 | 74.77 | 54.86 | 80.71 |
|
||||
| Zep | 88.11 | 71.99 | 74.45 | 66.67 | 81.06 |
|
||||
| TiMem | 81.43 | 62.20 | 77.63 | 52.08 | 75.30 |
|
||||
| TSM | 84.30 | 66.67 | 71.03 | 58.33 | 76.69 |
|
||||
| MemR3 | 89.44 | 71.39 | 76.22 | 61.11 | 81.55 |
|
||||
| **ReMe** | **89.89** | **82.98** | **83.80** | **71.88** | **86.23** |
|
||||
|
||||
|
||||
### HaluMem
|
||||
|
||||
| Method | Memory Integrity | Memory Accuracy | QA Accuracy |
|
||||
|-------------|------------------|---------------|-------------|
|
||||
| MemoBase | 14.55 | 92.24 | 35.53 |
|
||||
| Supermemory | 41.53 | 90.32 | 54.07 |
|
||||
| Mem0 | 42.91 | 86.26 | 53.02 |
|
||||
| ProMem | **73.80** | 89.47 | 62.26 |
|
||||
| **ReMe** | 67.72 | **94.06** | **88.78** |
|
||||
|
||||
|
||||
|
||||
### Python 使用
|
||||
|
||||
|
|
@ -550,8 +513,39 @@ graph LR
|
|||
|
||||
### 实验效果
|
||||
|
||||
Coming soon...
|
||||
本实验部分在 LoCoMo、HaluMem 两个数据集上进行评测，实验设置如下：
|
||||
|
||||
1. **ReMe 使用模型**：如各表 backbone 列所示。
|
||||
2. **评估使用模型**：采用 LLM-as-a-Judge 协议（参照 MemOS）——每条回答由 GPT-4o-mini 裁判模型打分。
|
||||
|
||||
实验设置尽量与各基线论文保持一致，以复用其公开结果。
|
||||
|
||||
|
||||
#### LoCoMo
|
||||
|
||||
| Method | Single Hop | Multi Hop | Temporal | Open Domain | Overall |
|
||||
|--------|------------|-----------|-----------|-------------|-----------|
|
||||
| MemoryOS | 62.43 | 56.50 | 37.18 | 40.28 | 54.70 |
|
||||
| Mem0 | 66.71 | 58.16 | 55.45 | 40.62 | 61.00 |
|
||||
| MemU | 72.77 | 62.41 | 33.96 | 46.88 | 61.15 |
|
||||
| MemOS | 81.45 | 69.15 | 72.27 | 60.42 | 75.87 |
|
||||
| HiMem | 89.22 | 70.92 | 74.77 | 54.86 | 80.71 |
|
||||
| Zep | 88.11 | 71.99 | 74.45 | 66.67 | 81.06 |
|
||||
| TiMem | 81.43 | 62.20 | 77.63 | 52.08 | 75.30 |
|
||||
| TSM | 84.30 | 66.67 | 71.03 | 58.33 | 76.69 |
|
||||
| MemR3 | 89.44 | 71.39 | 76.22 | 61.11 | 81.55 |
|
||||
| **ReMe** | **89.89** | **82.98** | **83.80** | **71.88** | **86.23** |
|
||||
|
||||
|
||||
#### HaluMem
|
||||
|
||||
| Method | Memory Integrity | Memory Accuracy | QA Accuracy |
|
||||
|-------------|------------------|---------------|-------------|
|
||||
| MemoBase | 14.55 | 92.24 | 35.53 |
|
||||
| Supermemory | 41.53 | 90.32 | 54.07 |
|
||||
| Mem0 | 42.91 | 86.26 | 53.02 |
|
||||
| ProMem | **73.80** | 89.47 | 62.26 |
|
||||
| **ReMe** | 67.72 | **94.06** | **88.78** |
|
||||
---
|
||||
|
||||
## 🧪 程序化记忆论文
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue