scriptoriumadmin/ReMe
1
0
Fork 0
mirror of https://github.com/agentscope-ai/ReMe.git synced 2026-09-15 23:31:05 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 9

docs(readme): 更新文档添加实验结果表格

Browse source 
- 在英文版 README 中添加 🧪 Experiments 章节
- 添加 LoCoMo 和 HaluMem 两个基准测试的结果表格
- 在中文版 README_ZH 中添加 🧪 实验 章节
- 添加 LoCoMo 和 HaluMem 测试集的实验配置说明
- 添加完整的实验数据对比表格和评估协议说明
This commit is contained in:
方应 2026-03-13 14:22:10 +08:00
parent 2f682b8d7c
commit 51eb53e831
2 changed files with 76 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

38
README.md
View file

@ -416,6 +416,44 @@ memories:
Installation and environment configuration are the same as [ReMeLight](#installation).
API keys are configured via environment variables and can be stored in a `.env` file at the project root.
## 🧪 Experiments
Evaluations are conducted on three benchmarks: **LoCoMo** and **HaluMem**. Experimental settings:
1. **ReMe backbone**: as specified in each table.
2. **Evaluation protocol**: LLM-as-a-Judge following MemOS — each answer is scored by GPT-4o-mini.
Baseline results are reproduced from their respective papers under aligned settings where possible.
### LoCoMo
| Method | Single Hop | Multi Hop | Temporal | Open Domain | Overall |
|--------|------------|-----------|-----------|-------------|-----------|
| MemoryOS | 62.43 | 56.50 | 37.18 | 40.28 | 54.70 |
| Mem0 | 66.71 | 58.16 | 55.45 | 40.62 | 61.00 |
| MemU | 72.77 | 62.41 | 33.96 | 46.88 | 61.15 |
| MemOS | 81.45 | 69.15 | 72.27 | 60.42 | 75.87 |
| HiMem | 89.22 | 70.92 | 74.77 | 54.86 | 80.71 |
| Zep | 88.11 | 71.99 | 74.45 | 66.67 | 81.06 |
| TiMem | 81.43 | 62.20 | 77.63 | 52.08 | 75.30 |
| TSM | 84.30 | 66.67 | 71.03 | 58.33 | 76.69 |
| MemR3 | 89.44 | 71.39 | 76.22 | 61.11 | 81.55 |
| **ReMe** | **89.89** | **82.98** | **83.80** | **71.88** | **86.23** |
### HaluMem
| Method | Memory Integrity | Memory Accuracy | QA Accuracy |
|-------------|------------------|---------------|-------------|
| MemoBase | 14.55 | 92.24 | 35.53 |
| Supermemory | 41.53 | 90.32 | 54.07 |
| Mem0 | 42.91 | 86.26 | 53.02 |
| ProMem | **73.80** | 89.47 | 62.26 |
| **ReMe** | 67.72 | **94.06** | **88.78** |
### Python usage
```python

38
README_ZH.md
View file

@ -396,6 +396,44 @@ graph LR
安装和环境变量配置与 [ReMeLight 一致](#安装)，通过环境变量设置 API 密钥，可写在项目根目录的 `.env` 文件中。
## 🧪 实验
本实验部分在 LoCoMo、LongMemEval、HaluMem 三个数据集上进行评测，实验设置如下：
1. **ReMe 使用模型**：如各表 backbone 列所示。
2. **评估使用模型**：采用 LLM-as-a-Judge 协议（参照 MemOS）——每条回答由 GPT-4o-mini 裁判模型打分。
实验设置尽量与各基线论文保持一致，以复用其公开结果。
### LoCoMo
| Method | Single Hop | Multi Hop | Temporal | Open Domain | Overall |
|--------|------------|-----------|-----------|-------------|-----------|
| MemoryOS | 62.43 | 56.50 | 37.18 | 40.28 | 54.70 |
| Mem0 | 66.71 | 58.16 | 55.45 | 40.62 | 61.00 |
| MemU | 72.77 | 62.41 | 33.96 | 46.88 | 61.15 |
| MemOS | 81.45 | 69.15 | 72.27 | 60.42 | 75.87 |
| HiMem | 89.22 | 70.92 | 74.77 | 54.86 | 80.71 |
| Zep | 88.11 | 71.99 | 74.45 | 66.67 | 81.06 |
| TiMem | 81.43 | 62.20 | 77.63 | 52.08 | 75.30 |
| TSM | 84.30 | 66.67 | 71.03 | 58.33 | 76.69 |
| MemR3 | 89.44 | 71.39 | 76.22 | 61.11 | 81.55 |
| **ReMe** | **89.89** | **82.98** | **83.80** | **71.88** | **86.23** |
### HaluMem
| Method | Memory Integrity | Memory Accuracy | QA Accuracy |
|-------------|------------------|---------------|-------------|
| MemoBase | 14.55 | 92.24 | 35.53 |
| Supermemory | 41.53 | 90.32 | 54.07 |
| Mem0 | 42.91 | 86.26 | 53.02 |
| ProMem | **73.80** | 89.47 | 62.26 |
| **ReMe** | 67.72 | **94.06** | **88.78** |
### Python 使用
```python