diff --git a/README.md b/README.md index ac54778c..0c488d8a 100644 --- a/README.md +++ b/README.md @@ -509,30 +509,31 @@ comprehensive [Quick Start Guide](./cookbook/simple_demo/quick_start.md). We test ExperienceMaker on Appworld with qwen3-8b: -| Method | best@1 | best@2 | best@4 | -|------------------------------------------|------------|--------------|------------| -| w/o ExperienceMaker (baseline) | 0.3561 | 0.4052 | 0.4536 | -| **w ExperienceMaker** | | | | -| [1] extract + compare + recall | **0.4069** | **0.5066** | 0.618 | -| [2] extract + compare + recall + rewrite | 0.3910 | 0.5038 | **0.6211** | +| Method | pass@1 | pass@2 | pass@4 | +|--------------------------------|-----------|-------------|-----------| +| w/o ExperienceMaker (baseline) | 0.083 | 0.140 | 0.228 | +| **w ExperienceMaker** | | | | +|experience(Direct Use) | **0.109** | **0.175** | **0.281** | -### 🔧 Experiment on BFCL-V3 +Pass@K measures the probability that at least one out of K generated samples successfully completes the task (achieves score=1). +The current experiments use an internal AppWorld environment which may have slight discrepancies, and we will soon update with experimental results from the standard AppWorld environment. + +You may find more details to reproduce this experiment in [quickstart.md](cookbook/appworld/quickstart.md) -Coming Soon! Stay tuned for comprehensive evaluation results. ### 🧊 Experiment on Frozenlake -| without experience | with experience | -|:-------------------------------------------:|:-------------------------------------------:| -|  |  | +| without experience | with experience | +|:----------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------:| +|

