diff --git a/benchmark/halumem/quickstart.md b/benchmark/halumem/quickstart.md new file mode 100644 index 00000000..59dc68fa --- /dev/null +++ b/benchmark/halumem/quickstart.md @@ -0,0 +1,33 @@ +# Halumem +Experiment Quick Start Guide +This guide helps you quickly set up and run Halumem experiments with ReMe integration. + +### 1. Start ReMe Service +Install ReMe (if not already installed) +If you haven't installed the ReMe environment yet, follow these steps: +```bash +# Create ReMe environment +conda create -p ./reme-env python==3.12 +conda activate ./reme-env + +# Install ReMe +pip install . +``` + +### 2. Clone the Repository +```bash +cd ./benchmark/halumem +git clone https://github.com/MemTensor/HaluMem.git +``` + +### 3. Run Experiments +Launch the ReMe service to enable memory library functionality: +```bash +clear && python benchmark/halumem/eval_reme.py \ + --data_path benchmark/halumem/HaluMem/data/HaluMem-Medium.jsonl \ + --reme_model_name gpt-4o-mini-2024-07-18 \ + --eval_model_name gpt-4o-mini-2024-07-18 \ + --batch_size 40 \ + --algo_version default +``` + diff --git a/benchmark/longmemeval/quickstart.md b/benchmark/longmemeval/quickstart.md new file mode 100644 index 00000000..c27d5c65 --- /dev/null +++ b/benchmark/longmemeval/quickstart.md @@ -0,0 +1,48 @@ +# Longmemeval +Experiment Quick Start Guide +This guide helps you quickly set up and run Longmemeval experiments with ReMe integration. + +### 1. Start ReMe Service +Install ReMe (if not already installed) +If you haven't installed the ReMe environment yet, follow these steps: +```bash +# Create ReMe environment +conda create -p ./reme-env python==3.12 +conda activate ./reme-env + +# Install ReMe +pip install . +``` + +### 2. Clone the Repository +```bash +cd ./benchmark/longmemeval +mkdir -p data/ +cd data/ +wget https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_oracle.json +wget https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json +wget https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_m_cleaned.json +cd .. +``` + +### 3. Run Experiments +Launch the ReMe service to enable memory library functionality: +```bash +clear && python benchmark/longmemeval/eval_longmemeval_reme.py \ + --data_path benchmark/longmemeval/data/longmemeval_s_cleaned.json \ + --reme_model_name qwen-flash \ + --reme_model_name retrieve_model_name \ + --eval_model_name gpt-4o-mini-2024-07-18 \ + --batch_size 20 \ + --algo_version default +``` + + +### 4. Evaluate Results +Evaluate the results of the experiments: +```bash +python benchmark/longmememeval/compute_stats.py \ + --results_dir bench_results/longmemeval_reme \ + --output_file bench_results/longmemeval_reme/statistics.json +``` +The `compute_stats.py` script computes various statistics from the evaluation results. \ No newline at end of file