ReMe/docs/cookbook/experiment_overview.md

Experiement Overview

🌍 Appworld Experiment

We tested ReMe on Appworld using qwen3-8b:

Method	pass@1	pass@2	pass@4
without ReMe	0.083	0.140	0.228
with ReMe	0.109 (+2.6%)	0.175 (+3.5%)	0.281 (+5.3%)

Pass@K measures the probability that at least one of the K generated samples successfully completes the task ( score=1). The current experiment uses an internal AppWorld environment, which may have slight differences.

You can find more details on reproducing the experiment in quickstart.md.

🧊 Frozenlake Experiment

without ReMe	with ReMe

We tested on 100 random frozenlake maps using qwen3-8b:

Method	pass rate
without ReMe	0.66
with ReMe	0.72 (+6.0%)

You can find more details on reproducing the experiment in quickstart.md.

🔧 BFCL-V3 Experiment

We tested ReMe on BFCL-V3 multi-turn-base (randomly split 50train/150val) using qwen3-8b:

Method	pass@1	pass@2	pass@4
without ReMe	0.2472	0.2733	0.2922
with ReMe	0.3061 (+5.89%)	0.3500 (+7.67%)	0.3888 (+9.66%)

🛠️ Tool Memory Benchmark

We evaluated Tool Memory effectiveness using a controlled benchmark with three mock search tools using Qwen3-30B-Instruct:

Scenario	Avg Score	Improvement
Train (No Memory)	0.650	-
Test (No Memory)	0.672	Baseline
Test (With Memory)	0.772	+14.88%

Key Findings:

• Tool Memory enables data-driven tool selection based on historical performance
• Success rates improved by ~15% with learned parameter configurations

You can find more details in tool_bench.md and the implementation at run_reme_tool_bench.py.