diff --git a/README.md b/README.md index 8f2f2762..1127d157 100644 --- a/README.md +++ b/README.md @@ -493,6 +493,32 @@ We test ExperienceMaker on Appworld with qwen3-8b: Coming Soon! Stay tuned for comprehensive evaluation results. +### 🧊 Experiment on Frozenlake +| without experience | with experience | +|:-------------------------------------------:|:-------------------------------------------:| +| ![GIF 1](doc/figure/frozenlake_failure.gif) | ![GIF 2](doc/figure/frozenlake_success.gif) | + +We test on 100 random frozenlake map with qwen3-8b: + +| Method | pass rate | +|-------------------------------|-------------------| +| w/o ExperienceMaker (baseline) | 0.66 | +| **w ExperienceMaker** | | +| [1] experience(Direct Use) | 0.84 **(+23.5%)** | +| [2] experience(LLM Rewritten) | 0.72 **(+9.1%)** | + +We also noticed that in such simple scenarios, not using LLM rewriting may actually yield better results. + +Therefore, in some simple scenarios, you can also try disabling LLM rewriting by simply changing the following in default_config.yaml: + +```yaml +rewrite_experience_op: + params: + enable_llm_rewrite: false # change this to false +``` + +You may find more details to reproduce this experiment quickly in `cookbook/frozenlake/quickstart.md` + --- ## 🏪 Ready-made Experience Store diff --git a/doc/figure/frozenlake_failure.gif b/doc/figure/frozenlake_failure.gif new file mode 100644 index 00000000..b9b7b8ca Binary files /dev/null and b/doc/figure/frozenlake_failure.gif differ diff --git a/doc/figure/frozenlake_success.gif b/doc/figure/frozenlake_success.gif new file mode 100644 index 00000000..3929d337 Binary files /dev/null and b/doc/figure/frozenlake_success.gif differ