mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-09-07 08:26:06 +00:00
d0c9d89092
* feat(memory): add ContextChecker component for context size management * refactor(memory): restructure file-based memory tools and update imports * docs(readme): update documentation with detailed architecture and components * docs(readme): update Chinese documentation with enhanced memory management diagrams * refactor(cookbook): move cookbook files to test directory and clean up docs * docs(readme): update link path for old version documentation * docs(readme): update documentation with improved architecture diagrams and component details * docs(readme): update documentation with improved clarity and structure * refactor(docs): update in-memory memory documentation * docs(readme): add experiment reproduction link to quickstart guide
129 lines
4.1 KiB
Markdown
129 lines
4.1 KiB
Markdown
# BFCL
|
|
Experiment Quick Start Guide
|
|
|
|
This guide helps you quickly set up and run BFCL experiments with ReMe integration.
|
|
|
|
## Env Setup
|
|
|
|
### 1. BFCL installation
|
|
|
|
#### Clone the repository
|
|
```bash
|
|
cd ReMe/benchmark/bfcl
|
|
git clone https://github.com/ShishirPatil/gorilla.git
|
|
cd gorilla
|
|
git checkout ea13468
|
|
```
|
|
|
|
#### Change directory to the `berkeley-function-call-leaderboard`
|
|
```bash
|
|
cd berkeley-function-call-leaderboard
|
|
```
|
|
|
|
#### Install the package in editable mode
|
|
```bash
|
|
pip install -e .
|
|
cd ../..
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
#### Move the dataset to the data folder under bfcl
|
|
```bash
|
|
cp -r gorilla/berkeley-function-call-leaderboard/bfcl_eval/data ./
|
|
```
|
|
|
|
#### Preprocess the data to get the suitable data format
|
|
```bash
|
|
python preprocess.py
|
|
```
|
|
|
|
**Note**: The original BFCL data is designed as a benchmark dataset and does not have a train/validation split, you can use ``split_into_trainval.py`` to split data into train and validation sets.
|
|
|
|
```bash
|
|
python split_into_trainval.py --input ./data/multiturn_data_base.jsonl --train ./data/multiturn_data_base_train.jsonl --val ./data/multiturn_data_base_val.jsonl
|
|
```
|
|
|
|
### 2. Start ReMe Service
|
|
|
|
After collecting trajectories, Launch the ReMe service (make sure you have installed ReMe environment, if not please follow the steps in the [ReMe Installation Guide](https://github.com/agentscope-ai/ReMe/blob/main/doc/README.md) to install):
|
|
|
|
```bash
|
|
reme2 \
|
|
backend=http \
|
|
http.port=8002 \
|
|
llms.default.model_name=qwen3-8b \
|
|
embedding_models.default.model_name=text-embedding-v4 \
|
|
vector_stores.default.backend=local \
|
|
vector_stores.default.collection_name=bfcl
|
|
```
|
|
|
|
<details>
|
|
<summary>Option: init the task memory pool from scratch</summary>
|
|
|
|
- First, collect agent trajectories on training data set without task memory:
|
|
|
|
```bash
|
|
# important: num_runs = 8, use_memory = False, experiment_suffix="wo-memory", data_path="data/multiturn_data_base_train.jsonl"
|
|
python run_bfcl.py
|
|
```
|
|
|
|
- Second, using ReMe to construct the initial task memory pool:
|
|
```bash
|
|
python init_task_memory_pool.py --jsonl_file ./exp_result/qwen3-8b/with_think/bfcl-multi-turn-base_wo-memory.jsonl
|
|
```
|
|
|
|
> Parameters:
|
|
> `jsonl_file`: Path to the collloaded trajectories
|
|
> `service_url`: ReMe service URL (default: `http://localhost:8002`)
|
|
> `n_threads`: Number of threads for processing
|
|
> `output_file`: Output file to save results (optional)
|
|
|
|
Now you have inited the task memory pool using `local` backend. Then, run the following `curl` command to dump the memory library:
|
|
```bash
|
|
curl -X POST "http://0.0.0.0:8002/dump_memory" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"dump_file_path": "./library/bfcl.jsonl",
|
|
}'
|
|
```
|
|
|
|
- Next time, you can import this previously exported task memory data to populate the new started workspace with existing knowledge:
|
|
```bash
|
|
curl -X POST "http://0.0.0.0:8002/load_memory" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"load_file_path": "./library/bfcl.jsonl",
|
|
"clear_existing": true
|
|
}'
|
|
```
|
|
</details>
|
|
|
|
### 3. Run Experiments on Validation Set
|
|
|
|
Run you can compare agent performance on the validation set with task memory (`use_memory=True`) and without task memory:
|
|
|
|
```bash
|
|
# remember to change the configuration options, e.g., `data_path=./data/multiturn_data_base_val.jsonl`
|
|
python run_bfcl.py
|
|
```
|
|
|
|
**Note**:
|
|
- `max_workers`: Number of parallel workers
|
|
- `num_runs`: Number of times each task is repeated
|
|
- `model_name`: LLM model name
|
|
- `enable_thinking`: Control the model's thinking mode
|
|
- `data_path`: Path to the training dataset (default: `./data/multiturn_data_base_val.jsonl`)
|
|
- `answer_path`: Path to the possible answer, which are used to evaluate the model's output function (default: `./data/possible_answer`)
|
|
- Results are automatically saved to `./exp_result/{model_name}/{no_think/with_think}` directory
|
|
|
|
After running experiments, analyze the statistical results:
|
|
|
|
```bash
|
|
python run_exp_statistic.py
|
|
```
|
|
|
|
**What this script does:**
|
|
- Processes all result files in `./exp_result/`
|
|
- Calculates best@k&pass@k metrics for different k values
|
|
- Generates a summary table showing performance comparisons
|
|
- Saves results to `experiment_summary.csv`
|