scriptoriumadmin/ReMe
1
0
Fork 0
mirror of https://github.com/agentscope-ai/ReMe.git synced 2026-08-28 05:25:04 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 11

update README.md reported results

Browse source
This commit is contained in:
caozouying.czy 2026-01-07 15:27:22 +08:00
parent 53aaad2a28
commit f493ba2f3a
2 changed files with 28 additions and 28 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

28
README.md
View file

@ -724,12 +724,12 @@ if __name__ == "__main__":
### 🌍 [Appworld Experiment](docs/cookbook/appworld/quickstart.md)
We tested ReMe on Appworld using qwen3-8b:
We tested ReMe on Appworld using Qwen3-8B (non-thinking mode):
| Method | pass@1 | pass@2 | pass@4 |
|--------------|-------------------|-------------------|-------------------|
| without ReMe | 0.083 | 0.140 | 0.228 |
| with ReMe | 0.109 **(+2.6%)** | 0.175 **(+3.5%)** | 0.281 **(+5.3%)** |
| Method | Avg@4 | Pass@4 |
|--------------|---------------------|---------------------|
| without ReMe | 0.1497 | 0.3285 |
| with ReMe | 0.1706 **(+2.09%)** | 0.3631 **(+3.46%)** |
Pass@K measures the probability that at least one of the K generated samples successfully completes the task (
score=1).
@ -737,6 +737,15 @@ The current experiment uses an internal AppWorld environment, which may have sli
You can find more details on reproducing the experiment in [quickstart.md](docs/cookbook/appworld/quickstart.md).
### 🔧 [BFCL-V3 Experiment](docs/cookbook/bfcl/quickstart.md)
We tested ReMe on BFCL-V3 multi-turn-base (randomly split 50train/150val) using Qwen3-8B (thinking mode):
| Method | Avg@4 | Pass@4 |
|--------------|---------------------|---------------------|
| without ReMe | 0.4033 | 0.5955 |
| with ReMe | 0.4450 **(+4.17%)** | 0.6577 **(+6.22%)** |
### 🧊 [Frozenlake Experiment](docs/cookbook/frozenlake/quickstart.md)
| without ReMe | with ReMe |
@ -752,15 +761,6 @@ We tested on 100 random frozenlake maps using qwen3-8b:
You can find more details on reproducing the experiment in [quickstart.md](docs/cookbook/frozenlake/quickstart.md).
### 🔧 [BFCL-V3 Experiment](docs/cookbook/bfcl/quickstart.md)
We tested ReMe on BFCL-V3 multi-turn-base (randomly split 50train/150val) using qwen3-8b:
| Method | pass@1 | pass@2 | pass@4 |
|--------------|---------------------|---------------------|---------------------|
| without ReMe | 0.2472 | 0.2733 | 0.2922 |
| with ReMe | 0.3061 **(+5.89%)** | 0.3500 **(+7.67%)** | 0.3888 **(+9.66%)** |
### 🛠️ [Tool Memory Benchmark](docs/tool_memory/tool_bench.md)
We evaluated Tool Memory effectiveness using a controlled benchmark with three mock search tools using Qwen3-30B-Instruct:

28
README_ZH.md
View file

@ -729,18 +729,27 @@ if __name__ == "__main__":
### 🌍 [Appworld 实验](docs/cookbook/appworld/quickstart.md)
我们在 Appworld 环境上使用 qwen3-8b 进行评测：
我们在 Appworld 环境上使用 Qwen3-8B（非思考模式）进行评测：
| 方法 | pass@1 | pass@2 | pass@4 |
|-----------|-------------------|-------------------|-------------------|
| 无 ReMe | 0.083 | 0.140 | 0.228 |
| 使用 ReMe | 0.109 **(+2.6%)** | 0.175 **(+3.5%)** | 0.281 **(+5.3%)** |
| 方法 | Avg@4 | Pass@4 |
|-----------|-------------------|-------------------|
| 无 ReMe | 0.1497 | 0.3285 |
| 使用 ReMe | 0.1706 **(+2.09%)** | 0.3631 **(+3.46%)** |
Pass@K 衡量在生成 K 个候选中，至少一个成功完成任务（score=1）的概率。
当前实验使用的是内部 AppWorld 环境，可能与对外版本存在轻微差异。
关于如何复现实验的更多细节，见 [quickstart.md](docs/cookbook/appworld/quickstart.md)。
### 🔧 [BFCL-V3 实验](docs/cookbook/bfcl/quickstart.md)
我们在 BFCL-V3 multi-turn-base 任务（随机划分 50 train / 150 val）上，使用 Qwen3-8B（思考模式）进行评测：
| 方法 | Avg@4 | Pass@4 |
|------------|-----------------|---------------------|
| 无 ReMe | 0.4033 | 0.5955 |
| 使用 ReMe | 0.4450 **(+4.17%)** | 0.6577 **(+6.22%)** |
### 🧊 [Frozenlake 实验](docs/cookbook/frozenlake/quickstart.md)
| 无 ReMe | 使用 ReMe |
@ -756,15 +765,6 @@ Pass@K 衡量在生成 K 个候选中，至少一个成功完成任务（score=1
更多复现实验细节见 [quickstart.md](docs/cookbook/frozenlake/quickstart.md)。
### 🔧 [BFCL-V3 实验](docs/cookbook/bfcl/quickstart.md)
我们在 BFCL-V3 multi-turn-base 任务（随机划分 50 train / 150 val）上，使用 qwen3-8b 进行评测：
| 方法 | pass@1 | pass@2 | pass@4 |
|------------|---------------------|---------------------|---------------------|
| 无 ReMe | 0.2472 | 0.2733 | 0.2922 |
| 使用 ReMe | 0.3061 **(+5.89%)** | 0.3500 **(+7.67%)** | 0.3888 **(+9.66%)** |
### 🛠️ [工具记忆基准](docs/tool_memory/tool_bench.md)
我们在一个受控基准上，使用三个模拟搜索工具与 Qwen3-30B-Instruct 评估工具记忆的效果：