mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-10-07 03:00:27 +00:00
docs(benchmark): add quick start guides for halumem and longmemeval experiments
- Created HaluMem experiment quick start guide with ReMe integration setup - Added detailed steps for installing ReMe environment using conda - Included repository cloning instructions for HaluMem benchmark - Provided complete command examples for running HaluMem experiments - Created LongMeMEval quick start guide with data download procedures - Added wget commands for downloading cleaned dataset files - Included evaluation script instructions for computing experiment statistics - Documented parameter configurations for different model types and batch sizes
This commit is contained in:
parent
949b3bf695
commit
89a86839a3
2 changed files with 81 additions and 0 deletions
33
benchmark/halumem/quickstart.md
Normal file
33
benchmark/halumem/quickstart.md
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
# Halumem
|
||||
Experiment Quick Start Guide
|
||||
This guide helps you quickly set up and run Halumem experiments with ReMe integration.
|
||||
|
||||
### 1. Start ReMe Service
|
||||
Install ReMe (if not already installed)
|
||||
If you haven't installed the ReMe environment yet, follow these steps:
|
||||
```bash
|
||||
# Create ReMe environment
|
||||
conda create -p ./reme-env python==3.12
|
||||
conda activate ./reme-env
|
||||
|
||||
# Install ReMe
|
||||
pip install .
|
||||
```
|
||||
|
||||
### 2. Clone the Repository
|
||||
```bash
|
||||
cd ./benchmark/halumem
|
||||
git clone https://github.com/MemTensor/HaluMem.git
|
||||
```
|
||||
|
||||
### 3. Run Experiments
|
||||
Launch the ReMe service to enable memory library functionality:
|
||||
```bash
|
||||
clear && python benchmark/halumem/eval_reme.py \
|
||||
--data_path benchmark/halumem/HaluMem/data/HaluMem-Medium.jsonl \
|
||||
--reme_model_name gpt-4o-mini-2024-07-18 \
|
||||
--eval_model_name gpt-4o-mini-2024-07-18 \
|
||||
--batch_size 40 \
|
||||
--algo_version default
|
||||
```
|
||||
|
||||
48
benchmark/longmemeval/quickstart.md
Normal file
48
benchmark/longmemeval/quickstart.md
Normal file
|
|
@ -0,0 +1,48 @@
|
|||
# Halumem
|
||||
Experiment Quick Start Guide
|
||||
This guide helps you quickly set up and run Halumem experiments with ReMe integration.
|
||||
|
||||
### 1. Start ReMe Service
|
||||
Install ReMe (if not already installed)
|
||||
If you haven't installed the ReMe environment yet, follow these steps:
|
||||
```bash
|
||||
# Create ReMe environment
|
||||
conda create -p ./reme-env python==3.12
|
||||
conda activate ./reme-env
|
||||
|
||||
# Install ReMe
|
||||
pip install .
|
||||
```
|
||||
|
||||
### 2. Clone the Repository
|
||||
```bash
|
||||
cd ./benchmark/longmemeval
|
||||
mkdir -p data/
|
||||
cd data/
|
||||
wget https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_oracle.json
|
||||
wget https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json
|
||||
wget https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_m_cleaned.json
|
||||
cd ..
|
||||
```
|
||||
|
||||
### 3. Run Experiments
|
||||
Launch the ReMe service to enable memory library functionality:
|
||||
```bash
|
||||
clear && python benchmark/longmemeval/eval_longmemeval_reme.py \
|
||||
--data_path benchmark/longmemeval/data/longmemeval_s_cleaned.json \
|
||||
--reme_model_name qwen-flash \
|
||||
--reme_model_name retrieve_model_name \
|
||||
--eval_model_name gpt-4o-mini-2024-07-18 \
|
||||
--batch_size 20 \
|
||||
--algo_version default
|
||||
```
|
||||
|
||||
|
||||
### 4. Evaluate Results
|
||||
Evaluate the results of the experiments:
|
||||
```bash
|
||||
python benchmark/longmememeval/compute_stats.py \
|
||||
--results_dir bench_results/longmemeval_reme \
|
||||
--output_file bench_results/longmemeval_reme/statistics.json
|
||||
```
|
||||
The `compute_stats.py` script computes various statistics from the evaluation results.
|
||||
Loading…
Add table
Reference in a new issue