|
+
+我们在 100 个随机 frozenlake 地图上使用 qwen3-8b 进行测试：
+
+| 方法 | pass rate |
+|---------------------|----------------|
+| 不使用 ReMe (baseline) | 0.66 |
+| **使用 ReMe** | |
+| w/ memory (直接使用) | 0.72 **(+9.1%)** |
+
+你可以在 [quickstart.md](cookbook/frozenlake/quickstart.md) 中找到复现实验的更多细节。
+
+### 🔧 BFCL-V3 实验
+
+即将推出！请持续关注完整的评估结果。
+
+## 📚 相关资源
+
+- **[快速开始](./cookbook/simple_demo)**：通过实际示例快速上手
+- **[向量存储设置](./doc/vector_store_api_guide.md)**：配置本地/向量数据库以及使用
+- **[mcp指南](./doc/mcp_quick_start.md)**：创建mcp服务
+- **链路说明**: 个性化记忆与任务记忆中分别使用的算子及其含义可以分别在 [personal memory](./doc/personal_memory) 与 [task memory](./doc/task_memory)中找到，你可以修改config以自定义链路
+- **[示例集合](./cookbook)**：实际用例和最佳实践
+
+---
+
+## 🤝 贡献
+
+我们相信最好的记忆系统来自集体智慧。欢迎贡献：
+
+### 代码贡献
+- 新操作和工具开发
+- 后端实现和优化
+- API增强和新端点
+
+### 文档改进
+- 使用示例和教程
+- 最佳实践指南
+
+---
+
+## 📄 引用
+
+```bibtex
+@software{ReMe2025,
+ title = {ReMe: Memory Framework for AI Agent},
+ author = {The ReMe Team},
+ url = {https://github.com/modelscope/ReMe},
+ year = {2025}
+}
+```
+
+---
+
+## ⚖️ 许可证
+
+本项目采用Apache License 2.0许可证 - 详情请参阅[LICENSE](./LICENSE)文件。
+
+---
\ No newline at end of file
diff --git a/cookbook/appworld/appworld_react_agent.py b/cookbook/appworld/appworld_react_agent.py
index 118e546b..19174350 100644
--- a/cookbook/appworld/appworld_react_agent.py
+++ b/cookbook/appworld/appworld_react_agent.py
@@ -36,9 +36,9 @@ class AppworldReactAgent:
max_interactions: int = 30,
max_response_size: int = 2048,
num_runs: int = 1,
- use_experience: bool = False,
- make_experience: bool = False,
- exp_url: str = "http://0.0.0.0:8001/",
+ use_task_memory: bool = False,
+ make_task_memory: bool = False,
+ api_url: str = "http://0.0.0.0:8002/",
workspace_id: str="appworld_v1"):
self.index: int = index
@@ -49,9 +49,9 @@ class AppworldReactAgent:
self.max_interactions: int = max_interactions
self.max_response_size: int = max_response_size
self.num_runs: int = num_runs
- self.use_experience: bool = use_experience
- self.make_experience: bool = make_experience
- self.exp_url = exp_url
+ self.use_task_memory: bool = use_task_memory
+ self.make_task_memory: bool = make_task_memory
+ self.api_url = api_url
self.workspace_id = workspace_id
self.llm_client = OpenAI()
@@ -75,10 +75,10 @@ class AppworldReactAgent:
return "call llm error"
def prompt_messages(self,world: AppWorld) -> list[dict]:
- if self.use_experience:
- experience = self.get_experience(world.task.instruction)
- logger.info(f"loaded experience: {experience}")
- dictionary = {"supervisor": world.task.supervisor, "instruction": world.task.instruction, "experience": experience}
+ if self.use_task_memory:
+ task_memory = self.get_task_memory(world.task.instruction)
+ logger.info(f"loaded task_memory: {task_memory}")
+ dictionary = {"supervisor": world.task.supervisor, "instruction": world.task.instruction, "experience": task_memory}
else:
dictionary = {"supervisor": world.task.supervisor, "instruction": world.task.instruction ,"experience": ""}
print(dictionary)
@@ -144,30 +144,75 @@ class AppworldReactAgent:
}
result.append(t_result)
- if self.make_experience:
- self.make_experience(result)
+ if self.make_task_memory:
+ memory_list = self.make_task_memory(result)
+ logger.info(f"Created {len(memory_list) if memory_list else 0} task memories")
return result
- def get_experience(self, query: str):
- response = requests.post(url=self.exp_url + "retriever", json={
- "workspace_id": self.workspace_id,
- "query": query,
- "top_k": 5
- })
-
+ def handle_api_response(self, response: requests.Response):
+ """Handle API response with proper error checking"""
if response.status_code != 200:
+ print(f"Error: {response.status_code}")
print(response.text)
+ return None
+
+ return response.json()
+
+ def get_task_memory(self, query: str):
+ """Retrieve relevant task memories based on a query"""
+ response = requests.post(
+ url=f"{self.api_url}retrieve_task_memory",
+ json={
+ "workspace_id": self.workspace_id,
+ "query": query,
+ }
+ )
+
+ result = self.handle_api_response(response)
+ if not result:
return ""
- response = response.json()
- print(response)
- experience_merged: str = response["experience_merged"]
- print(f"experience_merged={experience_merged}")
- return experience_merged
+ # Extract and return the answer
+ answer = result.get("answer", "")
+ print(f"Retrieved task memory: {answer}")
+ return answer
- def make_experience(self, result):
- pass
+ def make_task_memory(self, result):
+ """Generate a summary of conversation messages and create task memories"""
+ if not result:
+ print("No results to summarize")
+ return
+
+ # Prepare trajectories from results
+ trajectories = []
+ for r in result:
+ if "task_history" in r:
+ trajectories.append({
+ "messages": r["task_history"],
+ "score": float(r.get("uplift_score", 0.0))
+ })
+
+ if not trajectories:
+ print("No trajectories to summarize")
+ return
+
+ response = requests.post(
+ url=f"{self.api_url}summary_task_memory",
+ json={
+ "workspace_id": self.workspace_id,
+ "trajectories": trajectories
+ }
+ )
+
+ result = self.handle_api_response(response)
+ if not result:
+ return
+
+ # Extract memory list from response
+ memory_list = result.get("metadata", {}).get("memory_list", [])
+ print(f"Task memory list created: {len(memory_list)} memories")
+ return memory_list
def main():
diff --git a/cookbook/appworld/quickstart.md b/cookbook/appworld/quickstart.md
index 29a6910b..dd759eb3 100644
--- a/cookbook/appworld/quickstart.md
+++ b/cookbook/appworld/quickstart.md
@@ -1,14 +1,14 @@
# AppWorld Experiment Quick Start Guide
-This guide helps you quickly set up and run AppWorld experiments with ExperienceMaker integration.
+This guide helps you quickly set up and run AppWorld experiments with ReMe integration.
## Env Setup
### 1. Clone the Repository
```bash
-git clone https://github.com/modelscope/ExperienceMaker.git
-cd ExperienceMaker/cookbook/appworld
+git clone https://github.com/modelscope/ReMe.git
+cd ReMe/cookbook/appworld
```
### 2. Appworld Environment Setup
@@ -36,43 +36,43 @@ appworld download data
**Note**: The AppWorld data will be saved in the current directory.
-### 3. Start ExperienceMaker Service
+### 3. Start ReMe Service
-Install ExperienceMaker (if not already installed)
-If you haven't installed the ExperienceMaker environment yet, follow these steps:
+Install ReMe (if not already installed)
+If you haven't installed the ReMe environment yet, follow these steps:
```bash
# Go back to the project root
cd ../..
-# Create ExperienceMaker environment
-conda create -p ./em-env python==3.12
-conda activate ./em-env
+# Create ReMe environment
+conda create -p ./reme-env python==3.12
+conda activate ./reme-env
-# Install ExperienceMaker
+# Install ReMe
pip install .
```
-Launch the ExperienceMaker service to enable experience library functionality:
+Launch the ReMe service to enable memory library functionality:
```bash
-experiencemaker \
+reme \
http_service.port=8001 \
llm.default.model_name=qwen-max-latest \
embedding_model.default.model_name=text-embedding-v4 \
vector_store.default.backend=local_file
```
-add experiences for appworld:
+add memories for appworld:
```bash
curl -X POST "http://0.0.0.0:8001/vector_store" \
-H "Content-Type: application/json" \
-d '{
"workspace_id": "appworld_v1",
"action": "dump",
- "path": "./experience_library"
+ "path": "./memory_library"
}'
```
-Now you have loaded the ExperienceMaker experience library to enable experience-based agent!
+Now you have loaded the ReMe memory library to enable memory-based agent!
### 4. Common Issues
@@ -84,9 +84,9 @@ Now you have loaded the ExperienceMaker experience library to enable experience-
## Run Experiments
-### 1. Test: With Experience vs Without Experience
+### 1. Test: With Memory vs Without Memory
-Run the main experiment script to compare performance with and without experience:
+Run the main experiment script to compare performance with and without memory:
```bash
python run_appworld.py
@@ -94,7 +94,7 @@ python run_appworld.py
**What this does:**
- Runs AppWorld tasks on the development dataset
-- Compares agent performance with experience (`use_experience=True`) vs without experience
+- Compares agent performance with ReMe memory (`use_memory=True`) vs without memory
- Uses multiple workers for parallel processing
- Runs each task multiple times for statistical significance
- Results are automatically saved to `./exp_result/` directory
@@ -102,7 +102,7 @@ python run_appworld.py
**Configuration options in `run_appworld.py`:**
- `max_workers`: Number of parallel workers (default: 6)
- `num_runs`: Number of times each task is repeated (default: 4)
-- `use_experience`: Whether to use ExperienceMaker experience library
+- `use_memory`: Whether to use ReMe memory library
### 2. View Experiment Results
@@ -131,10 +131,10 @@ python run_exp_statistic.py
## Understanding Results
The experiment compares:
-1. **Baseline**: Agent without experience library
-2. **With Experience**: Agent enhanced with ExperienceMaker experience library
+1. **Baseline**: Agent without memory library
+2. **With Memory**: Agent enhanced with ReMe memory library
Key metrics to look for:
- **best@1**: Average performance across all single runs
- **best@k**: Performance when taking the best of k attempts
-- Improvement percentage when using experience vs baseline
\ No newline at end of file
+- Improvement percentage when using memory vs baseline
\ No newline at end of file
diff --git a/cookbook/appworld/run_appworld.py b/cookbook/appworld/run_appworld.py
index d16bcd9b..a3dcb7db 100644
--- a/cookbook/appworld/run_appworld.py
+++ b/cookbook/appworld/run_appworld.py
@@ -1,5 +1,6 @@
import os
import time
+import requests
import ray
from ray import logger
@@ -17,7 +18,64 @@ from appworld import load_task_ids
from appworld_react_agent import AppworldReactAgent
-def run_agent(dataset_name: str, experiment_suffix: str, max_workers: int, num_runs: int = 1, use_experience: bool = False, workspace_id: str="appworld", exp_url: str = "http://0.0.0.0:8001/") :
+def handle_api_response(response: requests.Response):
+ """Handle API response with proper error checking"""
+ if response.status_code != 200:
+ print(f"Error: {response.status_code}")
+ print(response.text)
+ return None
+
+ return response.json()
+
+
+def delete_workspace(workspace_id: str, api_url: str = "http://0.0.0.0:8002/"):
+ """Delete the current workspace from the vector store"""
+ response = requests.post(
+ url=f"{api_url}vector_store",
+ json={
+ "workspace_id": workspace_id,
+ "action": "delete",
+ }
+ )
+
+ result = handle_api_response(response)
+ if result:
+ print(f"Workspace '{workspace_id}' deleted successfully")
+
+
+def dump_memory(workspace_id: str, path: str = "./", api_url: str = "http://0.0.0.0:8002/"):
+ """Dump the vector store memories to disk"""
+ response = requests.post(
+ url=f"{api_url}vector_store",
+ json={
+ "workspace_id": workspace_id,
+ "action": "dump",
+ "path": path,
+ }
+ )
+
+ result = handle_api_response(response)
+ if result:
+ print(f"Memory dumped to {path}")
+
+
+def load_memory(workspace_id: str, path: str = "./", api_url: str = "http://0.0.0.0:8002/"):
+ """Load memories from disk into the vector store"""
+ response = requests.post(
+ url=f"{api_url}vector_store",
+ json={
+ "workspace_id": workspace_id,
+ "action": "load",
+ "path": path,
+ }
+ )
+
+ result = handle_api_response(response)
+ if result:
+ print(f"Memory loaded from {path}")
+
+
+def run_agent(dataset_name: str, experiment_suffix: str, max_workers: int, num_runs: int = 1, use_task_memory: bool = False, make_task_memory: bool = False, workspace_id: str="appworld_v1", api_url: str = "http://0.0.0.0:8002/") :
experiment_name = dataset_name + "_" + experiment_suffix
path: Path = Path(f"./exp_result")
path.mkdir(parents=True, exist_ok=True)
@@ -39,9 +97,10 @@ def run_agent(dataset_name: str, experiment_suffix: str, max_workers: int, num_r
task_ids=worker_task_ids,
experiment_name=experiment_name,
num_runs=num_runs,
- use_experience=use_experience,
+ use_task_memory=use_task_memory,
+ make_task_memory=make_task_memory,
workspace_id=workspace_id,
- exp_url=exp_url)
+ api_url=api_url)
future = actor.execute.remote()
future_list.append(future)
time.sleep(1)
@@ -64,7 +123,10 @@ def run_agent(dataset_name: str, experiment_suffix: str, max_workers: int, num_r
task_ids=[task_id],
experiment_name=experiment_name,
num_runs=num_runs,
- use_experience=use_experience)
+ use_task_memory=use_task_memory,
+ make_task_memory=make_task_memory,
+ workspace_id=workspace_id,
+ api_url=api_url)
task_results = agent.execute()
if isinstance(task_results, list):
result.extend(task_results)
@@ -75,18 +137,43 @@ def run_agent(dataset_name: str, experiment_suffix: str, max_workers: int, num_r
def main():
max_workers = 8
- num_runs = 1 # Run each task 4 times
+ num_runs = 1 # Run each task once
+ workspace_id = "appworld_v1"
+ api_url = "http://0.0.0.0:8002/"
+
if max_workers > 1:
ray.init(num_cpus=8)
-
- logger.info("Start running experiments without experience")
+
+ # Clean up workspace before starting
+ logger.info("Deleting workspace...")
+ delete_workspace(workspace_id=workspace_id, api_url=api_url)
+
+ # First run to build task memories
+ logger.info("Start running experiments to build task memories")
+ run_agent(dataset_name="dev", experiment_suffix="build-memory",
+ max_workers=max_workers, num_runs=1,
+ use_task_memory=False, make_task_memory=True,
+ workspace_id=workspace_id, api_url=api_url)
+
+ # Dump memories to disk for persistence
+ logger.info("Dumping memories to disk...")
+ dump_memory(workspace_id=workspace_id, api_url=api_url)
+
+ # Run experiments without task memory
+ logger.info("Start running experiments without task memory")
for i in range(num_runs):
- run_agent(dataset_name="dev", experiment_suffix=f"no-exp", max_workers=max_workers, num_runs=1,
- use_experience=False, workspace_id="appworld_v1")
+ run_agent(dataset_name="dev", experiment_suffix=f"no-memory",
+ max_workers=max_workers, num_runs=1,
+ use_task_memory=False, make_task_memory=False,
+ workspace_id=workspace_id, api_url=api_url)
- logger.info("Start running experiments with experience")
+ # Run experiments with task memory
+ logger.info("Start running experiments with task memory")
for i in range(num_runs):
- run_agent(dataset_name="dev", experiment_suffix=f"add-exp", max_workers=max_workers, num_runs=1, use_experience=True,workspace_id="appworld_v1")
+ run_agent(dataset_name="dev", experiment_suffix=f"with-memory",
+ max_workers=max_workers, num_runs=1,
+ use_task_memory=True, make_task_memory=False,
+ workspace_id=workspace_id, api_url=api_url)
diff --git a/cookbook/frozenlake/frozenlake_react_agent.py b/cookbook/frozenlake/frozenlake_react_agent.py
index 2e215db6..4f6538d5 100644
--- a/cookbook/frozenlake/frozenlake_react_agent.py
+++ b/cookbook/frozenlake/frozenlake_react_agent.py
@@ -32,7 +32,7 @@ class GameResult:
@ray.remote
class FrozenLakeReactAgent:
- """A ReAct Agent for FrozenLake game with experience learning."""
+ """A ReAct Agent for FrozenLake game with task memory learning."""
def __init__(self,
index: int,
@@ -42,8 +42,8 @@ class FrozenLakeReactAgent:
temperature: float = 0.7,
max_steps: int = 50,
num_runs: int = 1,
- use_experience: bool = False,
- make_experience: bool = False):
+ use_task_memory: bool = False,
+ make_task_memory: bool = False):
self.index = index
self.task_configs = task_configs
@@ -52,8 +52,8 @@ class FrozenLakeReactAgent:
self.temperature = temperature
self.max_steps = max_steps
self.num_runs = num_runs
- self.use_experience = use_experience
- self.make_experience = make_experience
+ self.use_task_memory = use_task_memory
+ self.make_task_memory = make_task_memory
self.llm_client = OpenAI()
self.action_map = {0: "LEFT", 1: "DOWN", 2: "RIGHT", 3: "UP"}
@@ -121,35 +121,34 @@ class FrozenLakeReactAgent:
else:
return self.prompts["frozenlake_sys_prompt_no_slippery"]
- def get_experience(self, map_desc: str, is_slippery: bool) -> str:
- """Retrieve relevant experience from experience service"""
- if not self.use_experience:
+ def get_task_memory(self, map_desc: str, is_slippery: bool) -> str:
+ """Retrieve relevant task memory from task memory service"""
+ if not self.use_task_memory:
return ""
try:
query = f"FrozenLake game map: {map_desc}, slippery: {is_slippery}"
- base_url = "http://0.0.0.0:8001/"
+ base_url = "http://0.0.0.0:8002/"
workspace_id = self.experiment_name
response = requests.post(
- url=base_url + "retriever",
+ url=base_url + "retrieve_task_memory",
json={
"workspace_id": workspace_id,
"query": query,
- "top_k": 3
},
timeout=60
)
if response.status_code == 200:
data = response.json()
- return data.get("experience_merged", "")
+ return data.get("answer", "")
else:
- logger.warning(f"Experience retrieval failed: {response.status_code}")
+ logger.warning(f"Task memory retrieval failed: {response.status_code}")
return ""
except Exception as e:
- logger.warning(f"Failed to get experience: {e}")
+ logger.warning(f"Failed to get task memory: {e}")
return ""
def action_parser(self, response: str) -> int:
@@ -194,19 +193,19 @@ class FrozenLakeReactAgent:
env = gym.make("FrozenLake-v1", **env_kwargs)
- # Get map description for experience
+ # Get map description for task memory
map_str = '\n'.join([''.join([cell.decode('utf-8') for cell in row])
for row in env.unwrapped.desc])
# Build messages
system_prompt = self.build_system_prompt(is_slippery)
- experience = self.get_experience(map_str, is_slippery)
+ task_memory = self.get_task_memory(map_str, is_slippery)
messages = [{"role": "system", "content": system_prompt}]
- if experience:
- exp_content = f"Here are some relevant tips from previous successful games:\n\n{experience}\n\nUse these tips to help you succeed."
- messages.append({"role": "user", "content": exp_content})
+ if task_memory:
+ memory_content = f"Here are some relevant tips from previous successful games:\n\n{task_memory}\n\nUse these tips to help you succeed."
+ messages.append({"role": "user", "content": memory_content})
messages.append(
{"role": "assistant", "content": "I'll use these tips to navigate the frozen lake successfully."})
@@ -279,56 +278,54 @@ class FrozenLakeReactAgent:
"map_id": map_id,
"is_slippery": is_slippery,
"map_size": map_size,
- "use_experience": self.use_experience
+ "use_task_memory": self.use_task_memory
}
)
return result, messages
- def save_experience(self, results: List[GameResult], messages_list: List[List[Dict]]):
- """Save successful trajectories as experience"""
- if not self.make_experience:
+ def save_task_memory(self, results: List[GameResult], messages_list: List[List[Dict]]):
+ """Save successful trajectories as task memory"""
+ if not self.make_task_memory:
return
- trajs = []
+ trajectories = []
for result, messages in zip(results, messages_list):
if result.success:
- # Create trajectory for experience service
+ # Create trajectory for task memory service
traj = {
"messages": messages,
- "query" : result.map_config["map_desc"],
"score": 1.0, # Success
}
- trajs.append(traj)
+ trajectories.append(traj)
else:
traj = {
"messages": messages,
- "query" : result.map_config["map_desc"],
- "score": 0.0, # Success
+ "score": 0.0, # Failure
}
- trajs.append(traj)
+ trajectories.append(traj)
- if trajs:
+ if trajectories:
try:
- base_url = "http://0.0.0.0:8001/"
+ base_url = "http://0.0.0.0:8002/"
workspace_id = self.experiment_name
response = requests.post(
- url=base_url + "summarizer",
+ url=base_url + "summary_task_memory",
json={
"workspace_id": workspace_id,
- "traj_list": trajs
+ "trajectories": trajectories
},
timeout=300
)
if response.status_code == 200:
- logger.info(f"Saved {len(trajs)} trajectories as experience")
+ logger.info(f"Saved {len(trajectories)} trajectories as task memory")
else:
- logger.warning(f"Failed to save experience: {response.status_code}")
+ logger.warning(f"Failed to save task memory: {response.status_code}")
except Exception as e:
- logger.error(f"Error saving experience: {e}")
+ logger.error(f"Error saving task memory: {e}")
def execute(self) -> List[Dict]:
"""Execute all tasks"""
@@ -357,9 +354,9 @@ class FrozenLakeReactAgent:
}
all_results[-1] = result_dict
- # Save experience if needed
- if self.make_experience:
- # Convert back to GameResult objects for experience saving
+ # Save task memory if needed
+ if self.make_task_memory:
+ # Convert back to GameResult objects for task memory saving
game_results = []
for i, result_dict in enumerate(all_results):
game_result = GameResult(
@@ -374,6 +371,6 @@ class FrozenLakeReactAgent:
)
game_results.append(game_result)
- self.save_experience(game_results, all_messages)
+ self.save_task_memory(game_results, all_messages)
return all_results
\ No newline at end of file
diff --git a/cookbook/frozenlake/quickstart.md b/cookbook/frozenlake/quickstart.md
index 40408b1a..bed7afaa 100644
--- a/cookbook/frozenlake/quickstart.md
+++ b/cookbook/frozenlake/quickstart.md
@@ -1,14 +1,14 @@
# FrozenLake Experiment Quick Start Guide
-This guide helps you quickly set up and run FrozenLake experiments with ExperienceMaker integration.
+This guide helps you quickly set up and run FrozenLake experiments with ReMe integration. The FrozenLake experiment demonstrates how task memory can improve an agent's performance in a navigation task.
-## Env Setup
+## Environment Setup
### 1. Clone the Repository
```bash
-git clone https://github.com/modelscope/ExperienceMaker.git
-cd ExperienceMaker/cookbook/frozenlake
+git clone https://github.com/modelscope/ReMe.git
+cd ReMe/cookbook/frozenlake
```
### 2. FrozenLake Environment Setup
@@ -19,49 +19,47 @@ Install Gymnasium for FrozenLake environment:
pip install gymnasium
```
-### 3. Start ExperienceMaker Service
+This will install:
+- gymnasium - for the FrozenLake environment
+- ray - for parallel execution
+- openai - for LLM API access
+- other dependencies
-Install ExperienceMaker (if not already installed)
-If you haven't installed the ExperienceMaker environment yet, follow these steps:
+### 3. Start ReMe Service
+
+If you haven't installed ReMe yet, follow these steps:
```bash
# Go back to the project root
cd ../..
-# Create ExperienceMaker environment
-conda create -p ./em-env python==3.12
-conda activate ./em-env
+# Create a virtual environment (optional)
+conda create -p ./reme-env python==3.10
+conda activate ./reme-env
-# Install ExperienceMaker
+# Install ReMe
pip install .
```
-Launch the ExperienceMaker service to enable experience library functionality:
+Launch the ReMe service to enable memory library functionality:
```bash
-experiencemaker \
- http_service.port=8001 \
- llm.default.model_name=qwen-max-latest \
+reme \
+ backend=http \
+ http.port=8002 \
+ llm.default.model_name=qwen-max-2025-01-25 \
embedding_model.default.model_name=text-embedding-v4 \
- vector_store.default.backend=local_file
+ vector_store.default.backend=local
```
-Load default experience library for FrozenLake:
+Load default memory library for FrozenLake:
```bash
-curl -X POST "http://0.0.0.0:8001/vector_store" \
- -H "Content-Type: application/json" \
- -d '{
- "workspace_id": "frozenlake_no_slippery",
- "action": "dump",
- "path": "./experience_library"
- }'
```
-Now you have loaded the default FrozenLake experience library to enable experience-based agent!
## Run Experiments
### 1. Quick Test: Performance Evaluation Only (Default)
-Run the main experiment script to test agent performance using existing experience:
+Run the main experiment script to test agent performance using existing memory:
```bash
python run_frozenlake.py
@@ -69,54 +67,33 @@ python run_frozenlake.py
**What this does:**
- Tests the agent on randomly generated FrozenLake maps
-- Uses the default experience library (`frozenlake_no_slippery`)
+- Uses the default memory library (`frozenlake_no_slippery`)
- Evaluates performance with multiple runs for statistical significance
- Results are automatically saved to `./exp_result/` directory
-### 2. Advanced: Training + Testing (Experience Generation)
+### 2. Advanced: Training + Testing (Memory Generation)
-To create new experiences through training and then test performance:
+To create new memories through training and then test performance:
-```bash
-python run_frozenlake.py --enable-training
+You can modify the experiment parameters directly in the `run_frozenlake.py` file. The main parameters are in the `main()` function:
+
+```python
+def main():
+ experiment_name = "frozenlake_no_slippery" # Name of the experiment
+ max_workers = 4 # Number of parallel workers
+ training_runs = 4 # Runs per training map
+ num_training_maps = 50 # Number of maps for training
+ test_runs = 1 # Runs per test configuration
+ num_test_maps = 100 # Number of test maps
+ is_slippery = False # Enable slippery mode
```
-**What this does:**
-- **Stage 1 (Training)**: Generates new experiences by solving training maps
-- **Stage 2 (Testing)**: Evaluates performance using the generated experiences
-- Compares baseline performance vs experience-enhanced performance
+Key parameters to consider:
+- `experiment_name`: Used as the workspace ID for task memory
+- `is_slippery`: When True, agent movement becomes stochastic (harder)
+- `max_workers`: Increase for faster execution on multi-core systems
-### 3. Custom Configuration Examples
-
-**Basic customization:**
-```bash
-python run_frozenlake.py --experiment-name "my_frozenlake_test" --max-workers 8
-```
-
-**Enable slippery mode:**
-```bash
-python run_frozenlake.py --slippery --experiment-name "frozenlake_slippery"
-```
-
-**Full training experiment:**
-```bash
-python run_frozenlake.py \
- --enable-training \
- --experiment-name "frozenlake_training_experiment" \
- --max-workers 8 \
- --training-runs 4 \
- --num-training-maps 50 \
- --test-runs 5 \
- --num-test-maps 100 \
- --slippery
-```
-
-**View all available options:**
-```bash
-python run_frozenlake.py --help
-```
-
-### 4. View Experiment Results
+### 3. View Experiment Results
After running experiments, analyze the statistical results:
@@ -128,30 +105,53 @@ python run_exp_statistic.py
- Processes all result files in `./exp_result/`
- Calculates success rates and performance metrics
- Generates a summary table showing performance comparisons
-- Saves results to `experiment_summary.csv`
+- Analyzes the effect of task memory on performance
+- Saves results to `frozenlake_summary.csv`
-## Configuration Parameters
+## Understanding the Implementation
-| Parameter | Default Value | Description |
-|-----------|---------------|-------------|
-| `--experiment-name` | `frozenlake_no_slippery` | Name of the experiment |
-| `--max-workers` | `4` | Number of parallel workers |
-| `--enable-training` | `False` | Enable training phase (experience generation) |
-| `--training-runs` | `4` | Number of runs per training map |
-| `--num-training-maps` | `50` | Number of training maps |
-| `--test-runs` | `1` | Number of runs per test configuration |
-| `--num-test-maps` | `100` | Number of test maps to use |
-| `--slippery` | `False` | Enable slippery ice mode |
+### Key Components
+
+1. **FrozenLakeReactAgent** (`frozenlake_react_agent.py`)
+ - Implements a ReAct agent that interacts with the FrozenLake environment
+ - Handles task memory retrieval and storage
+ - Uses LLM (via OpenAI API) for decision making
+
+2. **Experiment Runner** (`run_frozenlake.py`)
+ - Manages the overall experiment flow
+ - Handles training and testing phases
+ - Uses Ray for parallel execution
+
+3. **Map Manager** (`map_manager.py`)
+ - Generates and manages test maps
+ - Ensures consistent evaluation across experiments
+
+4. **Statistics Analyzer** (`run_exp_statistic.py`)
+ - Processes experiment results
+ - Calculates performance metrics
+ - Generates comparative analysis
## Understanding Results
The experiment evaluates agent performance on FrozenLake maps:
-- **Success Rate**: Percentage of episodes that reach the goal
-- **Default Mode**: Uses existing experience library for quick testing
-- **Training Mode**: Generates new experiences then tests performance improvement
+- **Success Rate**: Percentage of episodes where the agent reaches the goal
+- **With vs. Without Memory**: Compares performance with and without task memory
+- **Slippery vs. Non-slippery**: Compares performance in different environment dynamics
-**Output Files:**
-- `./exp_result/*.jsonl`: Raw experiment results
-- `./exp_result/experiment_summary.csv`: Statistical summary
-- Console output: Real-time progress and metrics
\ No newline at end of file
+### Output Files
+
+- `./exp_result/*_training.jsonl`: Results from training phase
+- `./exp_result/*_test_no_memory.jsonl`: Test results without task memory
+- `./exp_result/*_test_with_memory.jsonl`: Test results with task memory
+- `./exp_result/frozenlake_summary.csv`: Statistical summary
+
+### Task Memory Mechanism
+
+The task memory system works as follows:
+
+1. **Memory Creation**: During training, successful trajectories are sent to the ReMe service
+2. **Memory Retrieval**: During testing, the agent queries relevant memories based on the current map
+3. **Memory Application**: The agent uses retrieved memories to guide its decision-making
+
+The experiment demonstrates how task memory can significantly improve performance, especially in challenging environments like the slippery FrozenLake.
\ No newline at end of file
diff --git a/cookbook/frozenlake/run_frozenlake.py b/cookbook/frozenlake/run_frozenlake.py
index 7be7117c..66f30fb8 100644
--- a/cookbook/frozenlake/run_frozenlake.py
+++ b/cookbook/frozenlake/run_frozenlake.py
@@ -13,7 +13,7 @@ from map_manager import MapManager
def generate_training_configs(num_maps: int = 20, map_size: int = 4, is_slippery: bool=False) -> List[Dict]:
- """Generate random maps for training/experience generation"""
+ """Generate random maps for training/task memory generation"""
configs = []
for i in range(num_maps):
@@ -46,15 +46,15 @@ def generate_test_configs(num_test_maps: int = 100, is_slippery: bool = False) -
map_desc = np.array([list(row) for row in map_data["map_desc"]], dtype='c')
map_id = map_data["map_id"]
- for use_exp in [True, False]:
+ for use_memory in [True, False]:
config = {
"task_type": "test",
"map_desc": map_desc,
"map_size": 4,
"is_slippery": is_slippery,
- "use_experience": use_exp,
+ "use_task_memory": use_memory,
"map_id": map_id,
- "task_id": f"test_map{map_id}_slip{is_slippery}_exp{use_exp}"
+ "task_id": f"test_map{map_id}_slip{is_slippery}_mem{use_memory}"
}
configs.append(config)
@@ -63,8 +63,8 @@ def generate_test_configs(num_test_maps: int = 100, is_slippery: bool = False) -
def train(experiment_name: str, max_workers: int = 2, num_runs: int = 3, num_training_maps= 15, is_slippery: bool= False) -> None:
- """Phase 1: Generate experience from random maps"""
- logger.info("🎯 Starting Training Phase - Generating Experience")
+ """Phase 1: Generate task memory from random maps"""
+ logger.info("🎯 Starting Training Phase - Generating Task Memory")
logger.info("=" * 60)
training_configs = generate_training_configs(num_maps=num_training_maps, map_size=4, is_slippery=is_slippery)
@@ -91,8 +91,8 @@ def train(experiment_name: str, max_workers: int = 2, num_runs: int = 3, num_tra
task_configs=worker_configs,
experiment_name=experiment_name,
num_runs=num_runs,
- use_experience=False, # No experience in training phase
- make_experience=True,# Generate experience
+ use_task_memory=False, # No task memory in training phase
+ make_task_memory=True, # Generate task memory
)
future = agent.execute.remote()
future_list.append(future)
@@ -115,8 +115,8 @@ def train(experiment_name: str, max_workers: int = 2, num_runs: int = 3, num_tra
task_configs=training_configs,
experiment_name=experiment_name,
num_runs=num_runs,
- use_experience=False,
- make_experience=True
+ use_task_memory=False,
+ make_task_memory=True
)
results = agent.execute()
dump_results()
@@ -131,7 +131,7 @@ def train(experiment_name: str, max_workers: int = 2, num_runs: int = 3, num_tra
def test(experiment_name: str, max_workers: int = 2, num_runs: int = 5, num_test_maps: int = 100, is_slippery: bool=False) -> None:
- """Phase 2: Test on fixed maps with/without experience"""
+ """Phase 2: Test on fixed maps with/without task memory"""
logger.info("🧪 Starting Test Phase - Evaluating Performance")
logger.info(f"📊 Testing on {num_test_maps} maps with {num_runs} runs each")
logger.info("=" * 60)
@@ -140,12 +140,12 @@ def test(experiment_name: str, max_workers: int = 2, num_runs: int = 5, num_test
path = Path("./exp_result")
path.mkdir(parents=True, exist_ok=True)
- # Group configs by experience usage for separate experiments
- exp_configs = [c for c in test_configs if c.get("use_experience", False)]
- no_exp_configs = [c for c in test_configs if not c.get("use_experience", False)]
+ # Group configs by task memory usage for separate experiments
+ memory_configs = [c for c in test_configs if c.get("use_task_memory", False)]
+ no_memory_configs = [c for c in test_configs if not c.get("use_task_memory", False)]
- logger.info(f"📝 Configs without experience: {len(no_exp_configs)}")
- logger.info(f"📝 Configs with experience: {len(exp_configs)}")
+ logger.info(f"📝 Configs without task memory: {len(no_memory_configs)}")
+ logger.info(f"📝 Configs with task memory: {len(memory_configs)}")
@@ -156,37 +156,37 @@ def test(experiment_name: str, max_workers: int = 2, num_runs: int = 5, num_test
f.write(json.dumps(result) + "\n")
logger.info(f"💾 Test results saved to {output_file}")
- # Test without experience first
- logger.info("🚫 Testing WITHOUT experience...")
+ # Test without task memory first
+ logger.info("🚫 Testing WITHOUT task memory...")
all_results = []
- results_no_exp = run_test_configs(
- configs=no_exp_configs,
+ results_no_memory = run_test_configs(
+ configs=no_memory_configs,
experiment_name=experiment_name,
max_workers=max_workers,
num_runs=num_runs,
- use_experience=False
+ use_task_memory=False
)
- all_results.extend(results_no_exp)
- dump_results("no_exp")
+ all_results.extend(results_no_memory)
+ dump_results("no_memory")
- # Test with experience
- logger.info("✅ Testing WITH experience...")
+ # Test with task memory
+ logger.info("✅ Testing WITH task memory...")
all_results = []
- results_with_exp = run_test_configs(
- configs=exp_configs,
+ results_with_memory = run_test_configs(
+ configs=memory_configs,
experiment_name=experiment_name,
max_workers=max_workers,
num_runs=num_runs,
- use_experience=True
+ use_task_memory=True
)
- all_results.extend(results_with_exp)
- dump_results("with_exp")
+ all_results.extend(results_with_memory)
+ dump_results("with_memory")
return all_results
def run_test_configs(configs: List[Dict], experiment_name: str, max_workers: int,
- num_runs: int, use_experience: bool) -> List[Dict]:
+ num_runs: int, use_task_memory: bool) -> List[Dict]:
"""Run a set of test configurations"""
results = []
@@ -200,8 +200,8 @@ def run_test_configs(configs: List[Dict], experiment_name: str, max_workers: int
task_configs=worker_configs,
experiment_name=experiment_name,
num_runs=num_runs,
- use_experience=use_experience,
- make_experience=False
+ use_task_memory=use_task_memory,
+ make_task_memory=False
)
future = agent.execute.remote()
future_list.append(future)
@@ -219,8 +219,8 @@ def run_test_configs(configs: List[Dict], experiment_name: str, max_workers: int
task_configs=configs,
experiment_name=experiment_name,
num_runs=num_runs,
- use_experience=use_experience,
- make_experience=False
+ use_task_memory=use_task_memory,
+ make_task_memory=False
)
results = agent.execute()
@@ -258,8 +258,8 @@ def main():
is_slippery=is_slippery
)
- # Wait a bit for experience service to process
- logger.info("⏰ Waiting for experience service to process data...")
+ # Wait a bit for task memory service to process
+ logger.info("⏰ Waiting for task memory service to process data...")
time.sleep(10)
diff --git a/cookbook/simple_demo/use_personal_memory_demo.py b/cookbook/simple_demo/use_personal_memory_demo.py
index 80e3a650..231eebec 100644
--- a/cookbook/simple_demo/use_personal_memory_demo.py
+++ b/cookbook/simple_demo/use_personal_memory_demo.py
@@ -1,5 +1,6 @@
import asyncio
import json
+
import aiohttp
# API base URL
@@ -45,7 +46,9 @@ async def main():
async with session.post(
f"{base_url}/summary_personal_memory",
json={
- "messages": messages,
+ "trajectories": [
+ {"messages": messages, "score": 1.0}
+ ],
"workspace_id": workspace_id,
},
headers={"Content-Type": "application/json"}
diff --git a/cookbook/simple_demo/use_task_memory_mcp_demo.py b/cookbook/simple_demo/use_task_memory_mcp_demo.py
new file mode 100644
index 00000000..1a1e33a2
--- /dev/null
+++ b/cookbook/simple_demo/use_task_memory_mcp_demo.py
@@ -0,0 +1,287 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+Task Memory Demo for MemoryScope using MCP Client
+
+This script demonstrates how to use the task memory capabilities of MemoryScope
+through the MCP client interface. It shows how to run an agent, summarize conversations,
+retrieve memories, and manage the memory workspace.
+"""
+
+import json
+import time
+import asyncio
+from typing import List, Dict, Any, Optional
+
+from fastmcp import Client
+from mcp.types import CallToolResult
+from dotenv import load_dotenv
+
+# Load environment variables from .env file
+load_dotenv()
+
+# API configuration
+MCP_URL = "http://0.0.0.0:8002/sse/"
+WORKSPACE_ID = "test_workspace"
+
+
+async def delete_workspace(client: Client) -> None:
+ """
+ Delete the current workspace from the vector store
+
+ Args:
+ client: MCP client instance
+
+ Returns:
+ None
+ """
+ try:
+ result = await client.call_tool(
+ "vector_store",
+ arguments={
+ "workspace_id": WORKSPACE_ID,
+ "action": "delete",
+ }
+ )
+ print(f"Workspace '{WORKSPACE_ID}' deleted successfully")
+ except Exception as e:
+ print(f"Error deleting workspace: {e}")
+
+
+async def run_agent(client: Client, query: str, dump_messages: bool = False) -> List[Dict[str, Any]]:
+ """
+ Run the agent with a specific query
+
+ Args:
+ client: MCP client instance
+ query: The query to send to the agent
+ dump_messages: Whether to save messages to a file
+
+ Returns:
+ List of message objects from the conversation
+ """
+ try:
+ result = await client.call_tool(
+ "react",
+ arguments={"query": query}
+ )
+
+ # Extract and display the answer
+ response_data = json.loads(result.content)
+ answer = response_data.get("answer", "")
+ print(f"Agent response: {answer}")
+
+ # Get the conversation messages
+ messages = response_data.get("messages", [])
+
+ # Optionally save messages to file
+ if dump_messages and messages:
+ with open("messages.jsonl", "w") as f:
+ f.write(json.dumps(messages, indent=2, ensure_ascii=False))
+ print(f"Messages saved to messages.jsonl")
+
+ return messages
+ except Exception as e:
+ print(f"Error running agent: {e}")
+ return []
+
+
+async def run_summary(client: Client, messages: List[Dict[str, Any]], enable_dump_memory: bool = True) -> None:
+ """
+ Generate a summary of conversation messages and create task memories
+
+ Args:
+ client: MCP client instance
+ messages: List of message objects from a conversation
+ enable_dump_memory: Whether to save memory list to a file
+
+ Returns:
+ None
+ """
+ if not messages:
+ print("No messages to summarize")
+ return
+
+ try:
+ result = await client.call_tool(
+ "summary_task_memory",
+ arguments={
+ "workspace_id": WORKSPACE_ID,
+ "trajectories": [
+ {"messages": messages, "score": 1.0}
+ ]
+ }
+ )
+
+ response_data = json.loads(result.content)
+
+ # Extract memory list from response
+ memory_list = response_data.get("metadata", {}).get("memory_list", [])
+ print(f"Memory list: {memory_list}")
+
+ # Optionally save memory list to file
+ if enable_dump_memory and memory_list:
+ with open("task_memory.jsonl", "w") as f:
+ f.write(json.dumps(memory_list, indent=2, ensure_ascii=False))
+ print(f"Memory saved to task_memory.jsonl")
+ except Exception as e:
+ print(f"Error running summary: {e}")
+
+
+async def run_retrieve(client: Client, query: str) -> str:
+ """
+ Retrieve relevant task memories based on a query
+
+ Args:
+ client: MCP client instance
+ query: The query to retrieve relevant memories
+
+ Returns:
+ String containing the retrieved memory answer
+ """
+ try:
+ result = await client.call_tool(
+ "retrieve_task_memory",
+ arguments={
+ "workspace_id": WORKSPACE_ID,
+ "query": query,
+ }
+ )
+
+ response_data = json.loads(result.content)
+
+ # Extract and return the answer
+ answer = response_data.get("answer", "")
+ print(f"Retrieved memory: {answer}")
+ return answer
+ except Exception as e:
+ print(f"Error retrieving memory: {e}")
+ return ""
+
+
+async def run_agent_with_memory(client: Client, query_first: str, query_second: str, enable_dump_memory: bool = True) -> List[Dict[str, Any]]:
+ """
+ Run the agent with memory augmentation
+
+ This function demonstrates how to use task memory to enhance agent responses:
+ 1. First run the agent with the second query to build memory
+ 2. Then summarize the conversation to create memories
+ 3. Retrieve relevant memories for the first query
+ 4. Run the agent with the first query augmented with retrieved memories
+
+ Args:
+ client: MCP client instance
+ query_first: The query to run with memory augmentation
+ query_second: The query to build initial memories
+ enable_dump_memory: Whether to save memory list to a file
+
+ Returns:
+ List of message objects from the final conversation
+ """
+ # Run agent with second query to build initial memories
+ print(f"\n--- Building memories with query: '{query_second}' ---")
+ messages = await run_agent(client, query=query_second)
+
+ # Summarize conversation to create memories
+ print("\n--- Summarizing conversation to create memories ---")
+ await run_summary(client, messages, enable_dump_memory)
+ await asyncio.sleep(1)
+
+ # Retrieve relevant memories for the first query
+ print(f"\n--- Retrieving memories for query: '{query_first}' ---")
+ retrieved_memory = await run_retrieve(client, query_first)
+
+ # Run agent with first query augmented with retrieved memories
+ print(f"\n--- Running agent with memory-augmented query ---")
+ augmented_query = f"{retrieved_memory}\n\nUser Question:\n{query_first}"
+ print(f"Augmented query: {augmented_query}")
+ messages = await run_agent(client, query=augmented_query)
+
+ return messages
+
+
+async def dump_memory(client: Client, path: str = "./") -> None:
+ """
+ Dump the vector store memories to disk
+
+ Args:
+ client: MCP client instance
+ path: Directory path to save the memories
+
+ Returns:
+ None
+ """
+ try:
+ result = await client.call_tool(
+ "vector_store",
+ arguments={
+ "workspace_id": WORKSPACE_ID,
+ "action": "dump",
+ "path": path,
+ }
+ )
+ print(f"Memory dumped to {path}")
+ except Exception as e:
+ print(f"Error dumping memory: {e}")
+
+
+async def load_memory(client: Client, path: str = "./") -> None:
+ """
+ Load memories from disk into the vector store
+
+ Args:
+ client: MCP client instance
+ path: Directory path to load the memories from
+
+ Returns:
+ None
+ """
+ try:
+ result = await client.call_tool(
+ "vector_store",
+ arguments={
+ "workspace_id": WORKSPACE_ID,
+ "action": "load",
+ "path": path,
+ }
+ )
+ print(f"Memory loaded from {path}")
+ except Exception as e:
+ print(f"Error loading memory: {e}")
+
+
+async def main() -> None:
+ """
+ Main function to demonstrate task memory workflow
+ """
+ # Define example queries
+ query1 = "Analyze Xiaomi Corporation"
+ query2 = "Analyze the company Tesla."
+
+ print("=== Task Memory Demo (MCP Client) ===")
+
+ async with Client(MCP_URL) as client:
+ # Step 1: Clean up workspace
+ print("\n1. Deleting workspace...")
+ await delete_workspace(client)
+
+ # Step 2: Run agent with first query and save messages
+ print("\n2. Running agent with first query...")
+ await run_agent(client, query=query1, dump_messages=True)
+
+ # Step 3: Demonstrate memory-augmented agent
+ print("\n3. Running memory-augmented agent workflow...")
+ await run_agent_with_memory(client, query_first=query1, query_second=query2)
+
+ # Step 4: Demonstrate memory persistence
+ print("\n4. Dumping memory to disk...")
+ await dump_memory(client)
+
+ print("\n5. Loading memory from disk...")
+ await load_memory(client)
+
+ print("\n=== Demo Complete ===")
+
+
+if __name__ == "__main__":
+ asyncio.run(main())
diff --git a/doc/README.md b/doc/README.md
deleted file mode 100644
index 4c38397c..00000000
--- a/doc/README.md
+++ /dev/null
@@ -1,661 +0,0 @@
-# ExperienceMaker
-
-
-
-
-
-
-
-
-
-
-
-
-
- A comprehensive framework to make & reuse & share experience for AI agent
- Empowering agents to learn from the past and excel in the future
-
-
----
-
-## 📰 What's New
-- **[2025-08]** 🚀 MCP is now available! → [Quick Start Guide](./doc/mcp_quick_start.md)
-- **[2025-07]** 🎉 ExperienceMaker v0.1.1 is now available on [PyPI](https://pypi.org/project/experiencemaker/)!
-- **[2025-07]** 📚 Complete documentation and quick start guides released
-- **[2025-06]** 🚀 Multi-backend vector store support (Elasticsearch & ChromaDB)
-
----
-
-## 🚀 What's Next
-- **Pre-built Experience Libraries**: Domain repositories (Finance/Coding/Education/Research) + community marketplace
-- **Rich Experience Formats**: Executable code/tool configs/pipeline templates/workflows
-- **Experience Validation**: Quality analysis + cross-task effectiveness + auto-refinement
-- **Universal Trajectory Extraction**: Raw logs/multimodal data/execution traces → experiences
-
----
-
-## 🌟 What is ExperienceMaker?
-ExperienceMaker is a framework that transforms how AI agents learn and improve through **experience-driven intelligence**.
-By automatically extracting, storing, and intelligently reusing experiences from agent trajectories, it enables continuous learning and progressive skill enhancement.
-
-### ✨ Core Capabilities
-
-#### 🔍 **Intelligent Experience Summarizer**
-- **Success Pattern Recognition**: Identify what works and understand the underlying principles
-- **Failure Analysis**: Learn from mistakes to avoid repeating them in future tasks
-- **Comparative Insights**: Understand the critical differences between successful and failed approaches
-- **Multistep Trajectory Processing**: Break down complex tasks into learnable, actionable segments
-
-#### 🎯 **Smart Experience Retriever**
-- **Semantic Search**: Find relevant experiences using advanced embedding models and semantic understanding
-- **Context-Aware Ranking**: Prioritize the most applicable experiences for current task contexts
-- **Dynamic Rewriting**: Intelligently adapt experiences to fit new situations and requirements
-- **Multi-modal Support**: Handle various input types including query, messages
-
-#### 🗄️ **Scalable Experience Management**
-- **Multiple Storage Backends**: Choose from Elasticsearch (production-ready), ChromaDB (development), or file-based storage (testing)
-- **Workspace Isolation**: Organize experiences by projects, domains, or teams with complete separation
-- **Deduplication & Validation**: Ensure high-quality, unique experience storage with automated quality control
-- **Batch Operations**: Efficiently handle large-scale experience processing with optimized performance
-
-#### 🔧 **Developer-Friendly Architecture**
-- **REST API Interface**: Seamless integration with existing systems through clean API design
-- **Modular Pipeline Design**: Compose custom workflows from atomic operations with maximum flexibility
-- **Flexible Configuration**: YAML files and command-line overrides for easy customization
-- **Experience Store**: Ready-to-use out of the box — there’s no need for you to manually summarize experiences. You can directly leverage existing, comprehensive experience datasets to greatly enhance your agent’s capabilities.
-
-
-
-
----
-
-## 🛠️ Installation
-
-### Option 1: Install from PyPI (Recommended)
-
-```bash
-pip install experiencemaker
-```
-
-### Option 2: Install from Source
-
-```bash
-git clone https://github.com/modelscope/ExperienceMaker.git
-cd ExperienceMaker
-pip install .
-```
-
-## ⚙️ Environment Setup
-
-Create a `.env` file in your project root directory:
-
-```bash
-# Required: LLM API configuration
-LLM_API_KEY="sk-xxx"
-LLM_BASE_URL="https://xxx.com/v1"
-
-# Required: Embedding model configuration
-EMBEDDING_MODEL_API_KEY="sk-xxx"
-EMBEDDING_MODEL_BASE_URL="https://xxx.com/v1"
-
-# Optional: Elasticsearch configuration (if using Elasticsearch backend)
-
-```
-
-## 🚀 Quick Start
-
-### 🌐 HTTP Service
-
-For testing and development, use the `local_file` backend:
-```bash
-experiencemaker \
- http_service.port=8001 \
- llm.default.model_name=qwen3-32b \
- embedding_model.default.model_name=text-embedding-v4 \
- vector_store.default.backend=local_file
-```
-
-💡 **Pro Tip**: Check out our [Configuration Guide](./doc/configuration_guide.md) for detailed configuration topics
-including custom pipelines, operation parameters, and advanced configuration methods.
-
-The service will start on `http://localhost:8001`
-
-### 🔌 MCP Server
-
-ExperienceMaker now supports Model Context Protocol (MCP) for seamless integration with MCP-compatible clients like Claude Desktop:
-
-```bash
-experiencemaker_mcp \
- mcp_transport=stdio \
- llm.default.model_name=qwen3-32b \
- embedding_model.default.model_name=text-embedding-v4 \
- vector_store.default.backend=local_file
-```
-
-For SSE transport (Server-Sent Events):
-```bash
-experiencemaker_mcp \
- mcp_transport=sse \
- http_service.port=8001 \
- llm.default.model_name=qwen3-32b \
- embedding_model.default.model_name=text-embedding-v4 \
- vector_store.default.backend=local_file
-```
-
-🔗 **For detailed MCP setup and usage examples**, see our [MCP Quick Start Guide](./doc/mcp_quick_start.md).
-
-### 🔍 Production Setup with Elasticsearch Backend
-```bash
-experiencemaker \
- http_service.port=8001 \
- llm.default.model_name=qwen3-32b \
- embedding_model.default.model_name=text-embedding-v4 \
- vector_store.default.backend=elasticsearch
-```
-
-**Setup Elasticsearch:**
-```bash
-export ES_HOSTS="http://localhost:9200"
-# Quick setup using Elastic's official script
-curl -fsSL https://elastic.co/start-local | sh
-```
-📖 **Need Help?** Refer to [Vector Store Setup](./doc/vector_store_setup.md) for comprehensive deployment guidance.
-
-## 📝 Your First ExperienceMaker Script
-
-Here's how to get started!
-Note the `workspace_id` serves as your experience storage namespace. Experiences in different workspaces remain completely isolated and cannot access each other.
-
-### 📊 Call Summarizer Examples
-
-Transform conversation trajectories into valuable experiences using batch summarization. Each trajectory contains:
-
-- **Message**: Complete conversation history between user and agent
-- **Score**: Performance rating (0-1 scale, where 0=failure, 1=success)
-
-The summarizer analyzes these trajectories to extract actionable insights and patterns for future interactions.
-
-
-Python
-
-```python
-import requests
-
-response = requests.post(url="http://0.0.0.0:8001/summarizer", json={
- "workspace_id": "test_workspace",
- "traj_list": [
- {"messages": [{"role": "user", "content": "hello world"}], "score": 1.0}
- ]
-})
-
-experience_list = response.json()["experience_list"]
-for experience in experience_list:
- print(experience)
-```
-
-
-
-curl
-
-```bash
-curl -X POST "http://0.0.0.0:8001/summarizer" \
- -H "Content-Type: application/json" \
- -d '{
- "workspace_id": "test_workspace",
- "traj_list": [
- {
- "messages": [{"role": "user", "content": "hello world"}],
- "score": 1.0
- }
- ]
- }'
-```
-
-
-
-Node.js
-
-```javascript
-const fetch = require('node-fetch');
-// or: import fetch from 'node-fetch';
-
-async function callSummarizer() {
- try {
- const response = await fetch('http://0.0.0.0:8001/summarizer', {
- method: 'POST',
- headers: {
- 'Content-Type': 'application/json',
- },
- body: JSON.stringify({
- workspace_id: "test_workspace",
- traj_list: [
- {
- messages: [{ role: "user", content: "hello world" }],
- score: 1.0
- }
- ]
- })
- });
-
- const data = await response.json();
- const experienceList = data.experience_list;
-
- experienceList.forEach(experience => {
- console.log(experience);
- });
- } catch (error) {
- console.error('Error:', error);
- }
-}
-
-callSummarizer();
-```
-
-
-### 🔍 Call Retriever Examples
-
-Intelligently search and retrieve the most relevant experiences from your workspace to enhance decision-making. The retriever:
-
-- **Finds** the top-k most similar experiences based on semantic similarity to your query
-- **Returns** pre-assembled context ready for immediate use, or raw experience data for custom processing
-- **Leverages** your workspace's accumulated knowledge to provide contextually relevant insights
-
-
-Python
-
-```python
-import requests
-
-response = requests.post(url="http://0.0.0.0:8001/retriever", json={
- "workspace_id": "test_workspace",
- "query": "what is the meaning of life?",
- "top_k": 1,
-})
-
-experience_merged: str = response.json()["experience_merged"]
-print(f"experience_merged={experience_merged}")
-```
-
-
-
-curl
-
-```bash
-curl -X POST "http://0.0.0.0:8001/retriever" \
- -H "Content-Type: application/json" \
- -d '{
- "workspace_id": "test_workspace",
- "query": "what is the meaning of life?",
- "top_k": 1
- }'
-```
-
-
-
-Node.js
-
-```javascript
-const fetch = require('node-fetch');
-// or: import fetch from 'node-fetch';
-
-async function callRetriever() {
- try {
- const response = await fetch('http://0.0.0.0:8001/retriever', {
- method: 'POST',
- headers: {
- 'Content-Type': 'application/json',
- },
- body: JSON.stringify({
- workspace_id: "test_workspace",
- query: "what is the meaning of life?",
- top_k: 1
- })
- });
-
- const data = await response.json();
- const experienceMerged = data.experience_merged;
-
- console.log(`experience_merged=${experienceMerged}`);
- } catch (error) {
- console.error('Error:', error);
- }
-}
-
-callRetriever();
-```
-
-
-### 💾 Dump Experiences From Vector Store
-
-Export and backup your valuable experience data for archival, analysis, or migration purposes. This operation:
-
-- **Extracts** all experiences from the specified workspace in the vector store
-- **Saves** them to a structured JSONL file at `{path}/{workspace_id}.jsonl`
-- **Preserves** complete experience metadata and embeddings for future restoration
-
-
-Python
-
-```python
-import requests
-
-response = requests.post(url="http://0.0.0.0:8001/vector_store", json={
- "workspace_id": "test_workspace",
- "action": "dump",
- "path": "./",
-})
-print(response.json())
-```
-
-
-
-curl
-
-```bash
-curl -X POST "http://0.0.0.0:8001/vector_store" \
- -H "Content-Type: application/json" \
- -d '{
- "workspace_id": "test_workspace",
- "action": "dump",
- "path": "./"
- }'
-```
-
-
-
-Node.js
-
-```javascript
-const fetch = require('node-fetch');
-// or: import fetch from 'node-fetch';
-
-async function dumpExperiences() {
- try {
- const response = await fetch('http://0.0.0.0:8001/vector_store', {
- method: 'POST',
- headers: {
- 'Content-Type': 'application/json',
- },
- body: JSON.stringify({
- workspace_id: "test_workspace",
- action: "dump",
- path: "./"
- })
- });
-
- const data = await response.json();
- console.log(data);
- } catch (error) {
- console.error('Error:', error);
- }
-}
-
-dumpExperiences();
-```
-
-
-### 📥 Load Experiences To Vector Store
-
-Import and restore previously exported experience data to populate your workspace with existing knowledge. This operation:
-
-- **Reads** experience data from the JSONL file located at `{path}/{workspace_id}.jsonl`
-- **Reconstructs** the vector embeddings and indexes them in the specified workspace
-- **Enables** immediate access to imported experiences for retrieval and decision-making
-
-
-Python
-
-```python
-import requests
-
-response = requests.post(url="http://0.0.0.0:8001/vector_store", json={
- "workspace_id": "test_workspace",
- "action": "load",
- "path": "./",
-})
-
-print(response.json())
-```
-
-
-
-curl
-
-```bash
-curl -X POST "http://0.0.0.0:8001/vector_store" \
- -H "Content-Type: application/json" \
- -d '{
- "workspace_id": "test_workspace",
- "action": "load",
- "path": "./"
- }'
-```
-
-
-
-Node.js
-
-```javascript
-const fetch = require('node-fetch');
-// or: import fetch from 'node-fetch';
-
-async function loadExperiences() {
- try {
- const response = await fetch('http://0.0.0.0:8001/vector_store', {
- method: 'POST',
- headers: {
- 'Content-Type': 'application/json',
- },
- body: JSON.stringify({
- workspace_id: "test_workspace",
- action: "load",
- path: "./"
- })
- });
-
- const data = await response.json();
- console.log(data);
- } catch (error) {
- console.error('Error:', error);
- }
-}
-
-loadExperiences();
-```
-
-
-💡 **Need More Advanced Operations?** For additional workspace management features(e.g. delete_workspace,
-copy_workspace), advanced configuration options, and troubleshooting guidance, check out our
-comprehensive [Quick Start Guide](./cookbook/simple_demo/quick_start.md).
-
-🎭 **Want to See It in Action?** We've prepared a [simple react agent](./cookbook/simple_demo/simple_demo.py) that demonstrates how to enhance agent capabilities by integrating summarizer and retriever components, achieving significantly better performance.
-
----
-
-## 🧪 Experiments
-
-### 🌍 Experiment on Appworld
-
-We test ExperienceMaker on Appworld with qwen3-8b:
-
-| Method | pass@1 | pass@2 | pass@4 |
-|--------------------------------|-----------|-------------|-----------|
-| w/o ExperienceMaker (baseline) | 0.083 | 0.140 | 0.228 |
-| **w ExperienceMaker** | | | |
-|experience(Direct Use) | **0.109** | **0.175** | **0.281** |
-
-Pass@K measures the probability that at least one out of K generated samples successfully completes the task (achieves score=1).
-The current experiments use an internal AppWorld environment which may have slight discrepancies, and we will soon update with experimental results from the standard AppWorld environment.
-
-You may find more details to reproduce this experiment in [quickstart.md](cookbook/appworld/quickstart.md)
-
-
-### 🧊 Experiment on Frozenlake
-
-| without experience | with experience |
-|:-------------------------------------------------------------------------------------------:|:-------------------------------------------:|
-|
|
-
-We test on 100 random frozenlake map with qwen3-8b:
-
-| Method | pass rate |
-|-------------------------------|------------------|
-| w/o ExperienceMaker (baseline) | 0.66 |
-| **w ExperienceMaker** | |
-| [1] experience(Direct Use) | 0.72 **(+9.1%)** |
-| [2] experience(LLM Rewritten) | 0.72 **(+9.1%)** |
-
-We also noticed that in such simple scenarios, not using LLM rewriting may actually yield better results.
-
-Therefore, in some simple scenarios, you can also try disabling LLM rewriting by simply changing the following in default_config.yaml:
-
-```yaml
-rewrite_experience_op:
- params:
- enable_llm_rewrite: false # change this to false
-```
-
-You may find more details to reproduce this experiment in [quickstart.md](cookbook/frozenlake/quickstart.md)
-
-### 🔧 Experiment on BFCL-V3
-
-Coming Soon! Stay tuned for comprehensive evaluation results.
-
----
-
-## 🏪 Ready-made Experience Store
-
-ExperienceMaker provides pre-built experience libraries to jumpstart your agent's capabilities.
-You can directly load these curated experiences into your workspace and start benefiting from accumulated knowledge
-immediately.
-
-### 📦 Available Experience Libraries
-
-- **`appworld_v1.jsonl`**: Comprehensive experiences from Appworld agent interactions, covering complex task planning
- and execution patterns
-- **`bfcl_v1.jsonl`**: Function calling experiences from Berkeley Function-Calling Leaderboard tasks
-
-### 🚀 Quick Start with Pre-built Experiences
-
-Here's how to load and use the Appworld experience library:
-
-#### Step 1: Load Pre-built Experiences
-
-
-Python
-
-```python
-import requests
-
-# Load Appworld experiences into your workspace
-response = requests.post(url="http://0.0.0.0:8001/vector_store", json={
- "workspace_id": "appworld_v1",
- "action": "load",
- "path": "./library/",
-})
-
-print(f"loading result result={response.json()}")
-```
-
-
-
-curl
-
-```bash
-curl -X POST "http://0.0.0.0:8001/vector_store" \
- -H "Content-Type: application/json" \
- -d '{
- "workspace_id": "appworld_v1",
- "action": "load",
- "path": "./library/"
- }'
-```
-
-
-#### Step 2: Retrieve Relevant Experiences
-
-Now you can query the loaded experiences to get contextual guidance for your tasks:
-
-
-Python
-
-```python
-import requests
-
-# Query for app interaction experiences
-response = requests.post(url="http://0.0.0.0:8001/retriever", json={
- "workspace_id": "appworld_v1",
- "query": "How to navigate to settings and update user profile information?",
- "top_k": 1,
-})
-
-experience_merged = response.json()["experience_merged"]
-print(f"Retrieved experiences: {experience_merged}")
-```
-
-
-
-curl
-
-```bash
-curl -X POST "http://0.0.0.0:8001/retriever" \
- -H "Content-Type: application/json" \
- -d '{
- "workspace_id": "appworld_v1",
- "query": "How to navigate to settings and update user profile information?",
- "top_k": 1
- }'
-```
-
-
----
-
-## 📚 Additional Resources
-
-- **[Quick Start](./cookbook/simple_demo/quick_start.md)**: This guide will help you get started with ExperienceMaker quickly using practical examples.
-- **[Vector Store Setup](./doc/vector_store_setup.md)**: Complete production deployment guide
-- **[Configuration Guide](./doc/configuration_guide.md)**: Describes all available command-line parameters for ExperienceMaker Service
-- **[Operations Documentation](./doc/operations_documentation.md)**: Comprehensive operations configuration reference
-- **[Example Collection](./cookbook)**: Practical examples and use cases
-- **[Future RoadMap](./doc/future_roadmap.md)**: Our vision and upcoming features
-
----
-
-## 🤝 Contributing
-We warmly welcome contributions from the community! Here's how you can help make ExperienceMaker even better:
-
-### 🐛 **Report Issues**
-- Bug reports with detailed reproduction steps
-- Feature requests and enhancement suggestions
-- Documentation improvements and clarifications
-- Performance optimization ideas
-
-### 💻 **Code Contributions**
-- New operations and tools development
-- Backend implementations and optimizations
-- API enhancements and new endpoints
-- Test coverage improvements and quality assurance
-
-### 📝 **Documentation**
-- Usage examples and comprehensive tutorials
-- Best practices guides and design patterns
-- Translation and localization efforts
-
----
-## 📄 Citation
-If you use ExperienceMaker in your research or projects, please cite:
-```bibtex
-@software{ExperienceMaker,
- title = {ExperienceMaker: A Comprehensive Framework for AI Agent Experience Generation and Reuse},
- author = {The ExperienceMaker Team},
- url = {https://github.com/modelscope/ExperienceMaker},
- month = {08},
- year = {2025},
-}
-```
-
----
-## ⚖️ License
-This project is licensed under the Apache License 2.0 - see the [LICENSE](./LICENSE) file for details.
-
----
\ No newline at end of file
diff --git a/doc/ROADMAP.md b/doc/ROADMAP.md
deleted file mode 100644
index b3439151..00000000
--- a/doc/ROADMAP.md
+++ /dev/null
@@ -1,13 +0,0 @@
-1. library 转化 @zouyin
-2. index.html @jinli
-3. reme_ai两个personal的调通
-4. doc
- 1. readme @jiaji
- 2. experience maker @jiaji
- 3. personal @jinli
-5. 新增op @zouyin
-6. cookbook
- 1. appworld @jiaji P2
- 2. bfcl @zouyin P1
- 3. frozenlake @jiaji
- 4. simple_demo @jinli
\ No newline at end of file
diff --git a/doc/configuration_guide.md b/doc/configuration_guide.md
deleted file mode 100644
index 97670704..00000000
--- a/doc/configuration_guide.md
+++ /dev/null
@@ -1,337 +0,0 @@
-# Configuration Guide
-
-This document describes all available parameters for ExperienceMaker Service.
-The application uses [OmegaConf](https://omegaconf.readthedocs.io/) for configuration management, supporting both YAML
-files and command-line overrides.
-
-## Configuration Loading Priority
-
-1. Default values from `AppConfig` dataclass
-2. Pre-defined YAML configuration file (default: `demo_config.yaml`)
-3. Custom YAML file (if `config_path` is specified)
-4. Command-line overrides
-
-## 🏗️ Configuration Architecture
-
-ExperienceMaker uses a layered configuration system with the following priority order:
-
-1. **Default Configuration** (lowest priority)
-2. **YAML Configuration File**
-3. **Command Line Arguments** (highest priority)
-
-## Basic Bash Usage
-
-```bash
-experiencemaker [parameter1=value1] [parameter2=value2] ...
-```
-
-## 🧩 YAML Configuration Composition
-
-The YAML configuration file follows a specific composition pattern that enables flexible and modular configuration:
-
-### 1. Resource Declaration
-
-First, you declare the three core resources that form the foundation of the system:
-
-- **`llm`**: Language model configurations
-- **`embedding_model`**: Embedding model configurations
-- **`vector_store`**: Vector storage configurations
-
-In these sections, `default` (or any custom name) represents a declared configuration object that can be referenced
-later:
-
-```yaml
-llm:
- default: # This is a declared LLM configuration object
- backend: openai_compatible
- model_name: qwen3-32b
-
-embedding_model:
- default: # This is a declared embedding model configuration object
- backend: openai_compatible
- model_name: text-embedding-v4
-
-vector_store:
- default: # This is a declared vector store configuration object
- backend: local_file
- embedding_model: default
-```
-
-### 2. Operation Backend Registration
-
-In the `op` section, each operation declares its `backend` implementation. The backend names are registered through
-`@OP_REGISTRY.register()` decorator, typically converting camel-case class names to underscore format:
-
-```yaml
-op:
- recall_experience_op:
- backend: recall_experience_op # Registered via @OP_REGISTRY.register()
-```
-
-### 3. Resource References
-
-Operations reference the previously declared resources using their names:
-
-```yaml
-op:
- recall_experience_op:
- backend: recall_experience_op
- llm: default # References the declared LLM object
- embedding_model: default # References the declared embedding model object
- vector_store: default # References the declared vector store object
-```
-
-### 4. Pipeline
-
-Pipeline configurations use a special syntax to define operation flows:
-
-- `->`: Sequential execution
-- `[]`: Parallel execution group
-- `|`: Alternative operations within parallel group
-
-### Examples
-
-```yaml
-# Sequential pipeline
-api:
- retriever: op1->op2->op3
-
- # Parallel execution
- summarizer: op1->[op2|op3|op4]->op5
-
- # Complex pipeline with nested parallel operations
- vector_store: preprocess_op->[recall_op->rerank_op|backup_op]->merge_op
-```
-
-This compositional approach enables:
-
-- **Modularity**: Declare resources once, reference everywhere
-- **Flexibility**: Mix and match different backends and configurations
-- **Complexity**: Build sophisticated processing chains through pipeline syntax
-
-## 📁 Configuration Structure
-
-```yaml
-# Service Configuration
-http_service:
- host: "0.0.0.0"
- port: 8001
- timeout_keep_alive: 600
- limit_concurrency: 64
-
-# Pipeline Definitions
-api:
- retriever: recall_experience_op->rerank_experience_op->rewrite_experience_op
- summarizer: trajectory_preprocess_op->[success_extraction_op|failure_extraction_op]->experience_validation_op
- vector_store: vector_store_action_op
-
-# Operation Configurations
-op:
- operation_name:
- backend: operation_name # Register through `@OP_REGISTRY.register()`, typically by converting camel-cased types into underscored names
- llm: default # Optional: reference to LLM config, Register through `@LLM_REGISTRY.register()`
- embedding_model: default # Optional: reference to embedding config, Register through `@EMBEDDING_MODEL_REGISTRY.register()`
- vector_store: default # Optional: reference to vector store config, Register through `@VECTOR_STORE_REGISTRY.register()`
- params: # Operation-specific parameters
- param1: value1
- param2: value2
-
-# Resource Configurations
-llm:
- default:
- backend: openai_compatible
- model_name: qwen3-32b
- params:
- temperature: 0.6
-
-embedding_model:
- default:
- backend: openai_compatible
- model_name: text-embedding-v4
- params:
- dimensions: 1024
-
-vector_store:
- default:
- backend: local_file
- embedding_model: default
-```
-
-## Detailed Configuration Parameters
-
-| Parameter | Type | Default Value | Description | Example |
-|----------------------|--------|-----------------|----------------------------------------------------------------------|-------------------------------------------|
-| `pre_defined_config` | string | `"demo_config"` | Name of the pre-defined configuration file (without .yaml extension) | `pre_defined_config=full_pipeline_config` |
-| `config_path` | string | `""` | Path to custom configuration YAML file | `config_path=/path/to/config.yaml` |
-
-## HTTP Service Configuration
-
-| Parameter | Type | Default Value | Description | Example |
-|-----------------------------------|---------|---------------|-----------------------------------|---------------------------------------|
-| `http_service.host` | string | `"0.0.0.0"` | Host address for the HTTP service | `http_service.host=127.0.0.1` |
-| `http_service.port` | integer | `8001` | Port number for the HTTP service | `http_service.port=8080` |
-| `http_service.timeout_keep_alive` | integer | `600` | Keep-alive timeout in seconds | `http_service.timeout_keep_alive=600` |
-| `http_service.limit_concurrency` | integer | `64` | Maximum concurrent connections | `http_service.limit_concurrency=128` |
-
-## Thread Pool Configuration
-
-| Parameter | Type | Default Value | Description | Example |
-|---------------------------|---------|---------------|----------------------------------|------------------------------|
-| `thread_pool.max_workers` | integer | `10` | Maximum number of worker threads | `thread_pool.max_workers=20` |
-
-## API Pipeline Configuration
-
-| Parameter | Type | Default Value | Description | Example |
-|--------------------|--------|---------------|------------------------------------------|--------------------------------------------------------------|
-| `api.retriever` | string | `""` | Pipeline definition for retriever API | `api.retriever="build_query_op->recall_vector_store_op"` |
-| `api.summarizer` | string | `""` | Pipeline definition for summarizer API | `api.summarizer="simple_summary_op->update_vector_store_op"` |
-| `api.vector_store` | string | `""` | Pipeline definition for vector store API | `api.vector_store="vector_store_action_op"` |
-
-## Operation Configuration
-
-Operations are configured using the pattern `op.{operation_name}.{parameter}`. Each operation can have the following
-parameters:
-
-| Parameter | Type | Default Value | Description | Example |
-|------------------------------|--------|---------------|--------------------------------------------|------------------------------------------------------------------------------------------|
-| `op.{name}.backend` | string | `""` | Backend implementation class name | `op.build_query_op.backend=build_query_op` |
-| `op.{name}.prompt_file_path` | string | `""` | Path to prompt template file | `op.react_op.prompt_file_path=/path/to/prompt.yaml` |
-| `op.{name}.prompt_dict` | dict | `{}` | Direct prompt configuration dictionary | `op.react_op.prompt_dict.system="You are an AI assistant"` |
-| `op.{name}.llm` | string | `""` | Reference to LLM configuration | `op.react_op.llm=default` |
-| `op.{name}.embedding_model` | string | `""` | Reference to embedding model configuration | `op.recall_op.embedding_model=default` |
-| `op.{name}.vector_store` | string | `""` | Reference to vector store configuration | `op.recall_op.vector_store=default` |
-| `op.{name}.params.{param}` | any | `{}` | Operation-specific parameters | The parameter reference is in [operations_documentation.md](operations_documentation.md) |
-
-## LLM Configuration
-
-| Parameter | Type | Default Value | Description | Example |
-|-----------------------------|--------|---------------|----------------------------|-----------------------------------------|
-| `llm.{name}.backend` | string | `""` | LLM backend implementation | `llm.default.backend=openai_compatible` |
-| `llm.{name}.model_name` | string | `""` | Model name identifier | `llm.default.model_name=qwen3-32b` |
-| `llm.{name}.params.{param}` | any | `{}` | LLM-specific parameters | `llm.default.params.temperature=0.6` |
-
-## Embedding Model Configuration
-
-| Parameter | Type | Default Value | Description | Example |
-|-----------------------------------------|--------|---------------|----------------------------------------|--------------------------------------------------------|
-| `embedding_model.{name}.backend` | string | `""` | Embedding model backend implementation | `embedding_model.default.backend=openai_compatible` |
-| `embedding_model.{name}.model_name` | string | `""` | Embedding model name identifier | `embedding_model.default.model_name=text-embedding-v4` |
-| `embedding_model.{name}.params.{param}` | any | `{}` | Model-specific parameters | `embedding_model.default.params.dimensions=1024` |
-
-## Vector Store Configuration
-
-| Parameter | Type | Default Value | Description | Example |
-|---------------------------------------|--------|---------------|--------------------------------------------|-----------------------------------------------------------|
-| `vector_store.{name}.backend` | string | `""` | Vector store backend implementation | `vector_store.default.backend=elasticsearch` |
-| `vector_store.{name}.embedding_model` | string | `""` | Reference to embedding model configuration | `vector_store.default.embedding_model=default` |
-| `vector_store.{name}.params.{param}` | any | `{}` | Vector store-specific parameters | `vector_store.default.params.store_dir=file_vector_store` |
-
-
-## 🎯 Practical Examples
-
-### Example 1
-
-```bash
-experiencemaker \
- http_service.port=8002 \
- thread_pool.max_workers=64 \
- op.recall_experience_op.params.retrieve_top_k=50 \
- op.rerank_experience_op.params.top_k=10 \
- llm.default.params.temperature=0.1
-```
-
-### Example 2
-
-```yaml
-# dev_config.yaml
-http_service:
- port: 8003
-
-api:
- retriever: recall_experience_op->rerank_experience_op
-
-op:
- recall_experience_op:
- params:
- retrieve_top_k: 5 # Faster for development
-
- rerank_experience_op:
- params:
- top_k: 3
-
-llm:
- default:
- model_name: qwen-turbo
- params:
- temperature: 0.8
-```
-
-```bash
-experiencemaker config_path=dev_config.yaml
-```
-
-### Example 3: Multi-Backend Setup
-
-```yaml
-# multi_backend_config.yaml
-llm:
- fast:
- backend: openai_compatible
- model_name: qwen-turbo
- params:
- temperature: 0.9
-
- accurate:
- backend: openai_compatible
- model_name: gpt-4
- params:
- temperature: 0.1
-
-op:
- quick_extraction_op:
- backend: success_extraction_op
- llm: fast
-
- detailed_validation_op:
- backend: experience_validation_op
- llm: accurate
- params:
- validation_threshold: 0.8
-```
-
-## 📋 Configuration Tips
-
-1. **Start Simple**: Begin with the default configuration and override specific parameters
-2. **Use Environment Variables**: Set API keys and URLs in `.env` file
-3. **Parameter Validation**: Invalid parameters will cause startup errors with detailed messages
-4. **Performance Tuning**: Adjust `retrieve_top_k`, `top_k`, and `max_workers` based on your needs
-5. **Pipeline Testing**: Use simple pipelines first, then gradually add complexity
-
-## 🔍 Troubleshooting
-
-### Common Issues
-
-**Configuration Not Loading:**
-
-```bash
-# Check if config file exists and has correct YAML syntax
-experiencemaker config_path=/full/path/to/config.yaml
-```
-
-**Parameter Override Not Working:**
-
-```bash
-# Use exact parameter path from configuration structure
-experiencemaker op.operation_name.params.parameter_name=value
-```
-
-**Pipeline Syntax Errors:**
-
-- Check for balanced brackets `[]`
-- Ensure operation names exist in `op` section
-- Use `|` only within `[]` groups
-
----
-
-🎯 **Advanced Configuration Mastery!** You can now create sophisticated ExperienceMaker setups tailored to your specific
-needs.
\ No newline at end of file
diff --git a/doc/figure/framework.png b/doc/figure/framework.png
deleted file mode 100644
index d0b3a6d8..00000000
Binary files a/doc/figure/framework.png and /dev/null differ
diff --git a/doc/figure/logo.jpg b/doc/figure/logo.jpg
deleted file mode 100644
index 789b3e6d..00000000
Binary files a/doc/figure/logo.jpg and /dev/null differ
diff --git a/doc/figure/logo.png b/doc/figure/logo.png
deleted file mode 100644
index 8f548551..00000000
Binary files a/doc/figure/logo.png and /dev/null differ
diff --git a/doc/figure/logo_v2.png b/doc/figure/logo_v2.png
deleted file mode 100644
index 78decfdd..00000000
Binary files a/doc/figure/logo_v2.png and /dev/null differ
diff --git a/doc/figure/logo_v3.jpg b/doc/figure/logo_v3.jpg
deleted file mode 100644
index 782fee17..00000000
Binary files a/doc/figure/logo_v3.jpg and /dev/null differ
diff --git a/doc/figure/reme_logo.jpg b/doc/figure/reme_logo.jpg
new file mode 100644
index 00000000..e31779fc
Binary files /dev/null and b/doc/figure/reme_logo.jpg differ
diff --git a/doc/future_roadmap.md b/doc/future_roadmap.md
deleted file mode 100644
index e48ee57c..00000000
--- a/doc/future_roadmap.md
+++ /dev/null
@@ -1,59 +0,0 @@
-# 🗺️ ExperienceMaker Future Roadmap
-
-## P0 - Ready-to-Use Experience Libraries
-
-We aim to build curated experience libraries for complex scenarios, providing battle-tested best practices and lessons learned rather than simple documentation aggregation.
-
-Just as financial analysts develop analytical frameworks, senior engineers establish coding standards, and education experts create teaching methodologies, AI agents can build professional experience repositories. Start your AI projects standing on the shoulders of giants.
-
-**Core Features:**
-
-- [ ] Pre-built experience libraries for key domains
- - [ ] Finance
- - [ ] Coding
- - [ ] Education
- - [ ] Research
-- [ ] Experience marketplace: community-driven experience sharing and exchange
-
-## P0 - Support for Rich Experience Formats
-
-Expert knowledge extends beyond text to include debugged code, fine-tuned toolchains, and validated workflows. We aim to integrate diverse experience carriers:
-
-- [ ] **Executable Code**: Functions, code files, and scripts
-- [ ] **Tool Integration**: APIs, MCP configurations, and tool setups
-- [ ] **Pipeline Templates**: Agent execution pipelines and multi-step tool combinations
-
-## P0 - MCP Integration
-
-Modernize our API architecture by migrating three core APIs to the Model Context Protocol (MCP) standard for improved interoperability and standardization.
-
-- [ ] Summarizer API
-- [ ] Retriever API
-- [ ] Vector Store API
-
-## P1 - Experience Validation & Optimization
-
-AI-powered analysis of experience usage patterns and effectiveness, with automatic quality optimization and cross-task validation feedback loops.
-
-## P2 - Universal Trajectory Experience Extraction
-
-### Raw Data Processing
-- [ ] Automatic extraction of valuable experiences from agent execution logs
-- [ ] Multimodal support: images, videos, and other formats
-
-### Vision
-Transform valuable experience data from daily work into usable insights:
-- Communication techniques from emails
-- Optimization insights from code commits
-- Decision-making processes from meeting recordings
-
-Enable AI to naturally become stronger through everyday work, rather than wasting real-world experience data due to format limitations.
-
-## P2 - Open Source Experience Libraries
-
-Democratize AI experience sharing by making curated experience libraries publicly available on Hugging Face, enabling the broader AI community to benefit from and contribute to professional experience repositories.
-
-- [ ] **Hugging Face Integration**: Upload and maintain experience libraries on Hugging Face Hub
-- [ ] **Community Contributions**: Enable community-driven experience library improvements and additions
-- [ ] **Standardized Formats**: Establish standard formats for experience sharing across different domains
-- [ ] **Version Control**: Implement versioning system for experience library updates and improvements
diff --git a/doc/mcp_quick_start.md b/doc/mcp_quick_start.md
index d5acc7c8..aebef04d 100644
--- a/doc/mcp_quick_start.md
+++ b/doc/mcp_quick_start.md
@@ -1,15 +1,14 @@
-# ExperienceMaker MCP Quick Start Guide
+# MCP Quick Start Guide
-This guide will help you get started with ExperienceMaker using the Model Context Protocol (MCP) interface for seamless
+This guide will help you get started with ReMe using the Model Context Protocol (MCP) interface for seamless
integration with MCP-compatible clients.
## 🚀 What You'll Learn
-- How to set up ExperienceMaker MCP server
-- Connect to the server using MCP clients
-- Run an agent and generate experiences via MCP
-- Retrieve and apply experiences through MCP tools
-- Build experience-enhanced agents with MCP integration
+- How to set up and configure ReMe MCP server
+- How to connect to the server using Python MCP clients
+- How to use task memory operations through MCP
+- How to build experience-enhanced agents with MCP integration
## 📋 Prerequisites
@@ -23,14 +22,14 @@ integration with MCP-compatible clients.
### Option 1: Install from PyPI (Recommended)
```bash
-pip install experiencemaker
+pip install reme-ai
```
### Option 2: Install from Source
```bash
-git clone https://github.com/modelscope/ExperienceMaker.git
-cd ExperienceMaker
+git clone https://github.com/modelscope/ReMe.git
+cd ReMe
pip install .
```
@@ -39,77 +38,58 @@ pip install .
Create a `.env` file in your project directory:
```bash
-# Required: LLM API configuration
-LLM_API_KEY="sk-xxx"
-LLM_BASE_URL="https://xxx.com/v1"
+FLOW_EMBEDDING_API_KEY=sk-xxxx
+FLOW_EMBEDDING_BASE_URL=https://xxxx/v1
-# Required: Embedding model configuration
-EMBEDDING_MODEL_API_KEY="sk-xxx"
-EMBEDDING_MODEL_BASE_URL="https://xxx.com/v1"
-
-# Optional: Elasticsearch configuration (if using Elasticsearch backend)
-ES_HOSTS="http://localhost:9200"
+FLOW_LLM_API_KEY=sk-xxxx
+FLOW_LLM_BASE_URL=https://xxxx/v1
```
-## 🚀 Start the MCP Server
+## 🚀 Building an MCP Server with ReMe
-### Option 1: STDIO Transport (Recommended for MCP clients)
+ReMe provides a flexible framework for building MCP servers that can communicate using either STDIO or SSE (Server-Sent
+Events) transport protocols.
+
+### Starting the MCP Server
+
+#### Option 1: STDIO Transport (Recommended for MCP clients)
```bash
-experiencemaker_mcp \
- mcp_transport=stdio \
- llm.default.model_name=qwen3-32b \
+reme \
+ backend=mcp \
+ mcp.transport=stdio \
+ llm.default.model_name=qwen3-30b-a3b-thinking-2507 \
embedding_model.default.model_name=text-embedding-v4 \
- vector_store.default.backend=local_file
+ vector_store.default.backend=local
```
-### Option 2: SSE Transport (Server-Sent Events)
+#### Option 2: SSE Transport (Server-Sent Events)
```bash
-experiencemaker_mcp \
- mcp_transport=sse \
+reme \
+ backend=mcp \
+ mcp.transport=sse \
http_service.port=8001 \
- llm.default.model_name=qwen3-32b \
+ llm.default.model_name=qwen3-30b-a3b-thinking-2507 \
embedding_model.default.model_name=text-embedding-v4 \
- vector_store.default.backend=local_file
+ vector_store.default.backend=local
```
-The SSE server will start on `http://localhost:8001/sse`
+The SSE server will start on `http://localhost:8002/sse`
-### Elasticsearch Backend
+### Configuring MCP Server for Claude Desktop
-```bash
-experiencemaker_mcp \
- mcp_transport=stdio \
- llm.default.model_name=qwen3-32b \
- embedding_model.default.model_name=text-embedding-v4 \
- vector_store.default.backend=elasticsearch
-```
-
-**Setup Elasticsearch:**
-
-```bash
-export ES_HOSTS="http://localhost:9200"
-# Quick setup using Elastic's official script
-curl -fsSL https://elastic.co/start-local | sh
-```
-
-📖 **Need Help?** Refer to [Vector Store Setup](vector_store_setup.md) for comprehensive deployment guidance.
-
-## 🔧 Configure MCP Client
-
-### Claude Desktop Configuration
-
-Add to your Claude Desktop `claude_desktop_config.json`:
+To integrate with Claude Desktop, add the following configuration to your `claude_desktop_config.json`:
```json
{
"mcpServers": {
- "experiencemaker": {
- "command": "experiencemaker_mcp",
+ "reme": {
+ "command": "reme",
"args": [
- "mcp_transport=stdio",
- "llm.default.model_name=qwen3-32b",
+ "backend=mcp",
+ "mcp.transport=stdio",
+ "llm.default.model_name=qwen3-30b-a3b-thinking-2507",
"embedding_model.default.model_name=text-embedding-v4",
"vector_store.default.backend=local_file"
]
@@ -118,453 +98,311 @@ Add to your Claude Desktop `claude_desktop_config.json`:
}
```
-### Custom MCP Client Configuration
+This configuration:
-If using a custom MCP client, connect to:
+1. Registers a new MCP server named "reme"
+2. Specifies the command to launch the server (`reme`)
+3. Configures the server to use STDIO transport
+4. Sets the LLM and embedding models to use
+5. Configures the vector store backend
-- **STDIO**: Use subprocess to communicate with the server
-- **SSE**: Connect to `http://localhost:8001/sse`
+### Advanced Server Configuration Options
-## 📝 Using ExperienceMaker MCP Tools
+For more advanced use cases, you can configure the server with additional parameters:
-The MCP server exposes three main tools:
+```bash
+# Full configuration example
+reme \
+ backend=mcp \
+ mcp.transport=stdio \
+ http_service.host=0.0.0.0 \
+ http_service.port=8002 \
+ llm.default.model_name=qwen3-30b-a3b-thinking-2507 \
+ embedding_model.default.model_name=text-embedding-v4 \
+ vector_store.default.backend=elasticsearch \
+```
-- `retriever`: Retrieve experiences from workspace
-- `summarizer`: Transform trajectories into experiences
-- `vector_store`: Manage vector store operations
+## 🔌 Using Python Client to Call MCP Services
-Note: The `workspace_id` serves as your experience storage namespace. Experiences in different workspaces remain
-completely isolated.
+The ReMe framework provides a Python client for interacting with MCP services. This section focuses specifically on
+using the `summary_task_memory` and `retrieve_task_memory` tools.
-### 📊 Using the Summarizer Tool
+### Setting Up the Python MCP Client
-Transform conversation trajectories into valuable experiences using batch summarization.
+First, install the required packages:
-**Tool Parameters:**
+```bash
+pip install fastmcp dotenv
+```
-- `traj_list`: List of trajectories (each containing messages and score)
-- `workspace_id`: Workspace identifier (default: "default")
-- `config`: Additional configuration parameters (optional)
-
-
-Python MCP Client Example
+Then, create a basic client connection:
```python
import asyncio
+from fastmcp import Client
+from dotenv import load_dotenv
-from experiencemaker.schema.message import Message, Trajectory, Role
-from experiencemaker.schema.request import SummarizerRequest
-from experiencemaker.service.mcp_client import MCPClient
+# Load environment variables
+load_dotenv()
+
+# MCP server URL (for SSE transport)
+MCP_URL = "http://0.0.0.0:8002/sse/"
+WORKSPACE_ID = "my_workspace"
-async def example_summarizer():
- async with MCPClient(base_url="http://0.0.0.0:8001/sse") as client:
- # Create trajectory with conversation
- trajectory = Trajectory(
- messages=[
- Message(role=Role.USER, content="Hello, how can I solve a math problem?"),
- Message(role=Role.ASSISTANT, content="I'd be happy to help! What math problem are you working on?"),
- Message(role=Role.USER, content="What is 2+2?"),
- Message(role=Role.ASSISTANT, content="2+2 equals 4.")
- ],
- score=1.0 # Success score
- )
-
- request = SummarizerRequest(
- workspace_id="math_workspace",
- traj_list=[trajectory]
- )
-
- response = await client.call_summarizer(request)
- print("Generated experiences:")
- for experience in response.experience_list:
- print(f"- {experience.content}")
+async def main():
+ async with Client(MCP_URL) as client:
+ # Your MCP operations will go here
+ pass
-# Run the example
-asyncio.run(example_summarizer())
+if __name__ == "__main__":
+ asyncio.run(main())
```
-
+### Using the Task Memory Summarizer
-
-MCP Tool Call (JSON)
+The `summary_task_memory` tool transforms conversation trajectories into valuable task memories:
-```json
-{
- "method": "tools/call",
- "params": {
- "name": "summarizer",
- "arguments": {
- "traj_list": [
- {
- "messages": [
- {
- "role": "user",
- "content": "Hello, how can I solve a math problem?"
- },
- {
- "role": "assistant",
- "content": "I'd be happy to help! What math problem are you working on?"
- },
- {
- "role": "user",
- "content": "What is 2+2?"
- },
- {
- "role": "assistant",
- "content": "2+2 equals 4."
+```python
+async def run_summary(client, messages):
+ """
+ Generate a summary of conversation messages and create task memories
+
+ Args:
+ client: MCP client instance
+ messages: List of message objects from a conversation
+
+ Returns:
+ None
+ """
+ try:
+ result = await client.call_tool(
+ "summary_task_memory",
+ arguments={
+ "workspace_id": "my_workspace",
+ "trajectories": [
+ {"messages": messages, "score": 1.0}
+ ]
}
- ],
- "score": 1.0
+ )
+
+ # Parse the response
+ import json
+ response_data = json.loads(result.content)
+
+ # Extract memory list from response
+ memory_list = response_data.get("metadata", {}).get("memory_list", [])
+ print(f"Created memories: {memory_list}")
+
+ # Optionally save memories to file
+ with open("task_memory.jsonl", "w") as f:
+ f.write(json.dumps(memory_list, indent=2, ensure_ascii=False))
+
+ except Exception as e:
+ print(f"Error running summary: {e}")
+```
+
+### Using the Task Memory Retriever
+
+The `retrieve_task_memory` tool allows you to retrieve relevant memories based on a query:
+
+```python
+async def run_retrieve(client, query):
+ """
+ Retrieve relevant task memories based on a query
+
+ Args:
+ client: MCP client instance
+ query: The query to retrieve relevant memories
+
+ Returns:
+ String containing the retrieved memory answer
+ """
+ try:
+ result = await client.call_tool(
+ "retrieve_task_memory",
+ arguments={
+ "workspace_id": "my_workspace",
+ "query": query,
+ }
+ )
+
+ # Parse the response
+ import json
+ response_data = json.loads(result.content)
+
+ # Extract and return the answer
+ answer = response_data.get("answer", "")
+ print(f"Retrieved memory: {answer}")
+ return answer
+
+ except Exception as e:
+ print(f"Error retrieving memory: {e}")
+ return ""
+```
+
+### Complete Memory-Augmented Agent Example
+
+Here's a complete example showing how to build a memory-augmented agent using the MCP client:
+
+```python
+import json
+import asyncio
+from fastmcp import Client
+from dotenv import load_dotenv
+
+# Load environment variables
+load_dotenv()
+
+# API configuration
+MCP_URL = "http://0.0.0.0:8002/sse/"
+WORKSPACE_ID = "test_workspace"
+
+
+async def run_agent(client, query):
+ """Run the agent with a specific query"""
+ result = await client.call_tool(
+ "react",
+ arguments={"query": query}
+ )
+
+ response_data = json.loads(result.content)
+ answer = response_data.get("answer", "")
+ messages = response_data.get("messages", [])
+
+ return messages
+
+
+async def run_summary(client, messages):
+ """Generate task memories from conversation"""
+ result = await client.call_tool(
+ "summary_task_memory",
+ arguments={
+ "workspace_id": WORKSPACE_ID,
+ "trajectories": [
+ {"messages": messages, "score": 1.0}
+ ]
}
- ],
- "workspace_id": "math_workspace"
- }
- }
-}
+ )
+
+ response_data = json.loads(result.content)
+ memory_list = response_data.get("metadata", {}).get("memory_list", [])
+
+ return memory_list
+
+
+async def run_retrieve(client, query):
+ """Retrieve relevant task memories"""
+ result = await client.call_tool(
+ "retrieve_task_memory",
+ arguments={
+ "workspace_id": WORKSPACE_ID,
+ "query": query,
+ }
+ )
+
+ response_data = json.loads(result.content)
+ answer = response_data.get("answer", "")
+
+ return answer
+
+
+async def memory_augmented_workflow():
+ """Complete memory-augmented agent workflow"""
+ query1 = "Analyze Xiaomi Corporation"
+ query2 = "Analyze the company Tesla."
+
+ async with Client(MCP_URL) as client:
+ # Step 1: Build initial memories with query2
+ print(f"Building memories with: '{query2}'")
+ messages = await run_agent(client, query=query2)
+
+ # Step 2: Summarize conversation to create memories
+ print("Creating memories from conversation")
+ memory_list = await run_summary(client, messages)
+ print(f"Created {len(memory_list)} memories")
+
+ # Step 3: Retrieve relevant memories for query1
+ print(f"Retrieving memories for: '{query1}'")
+ retrieved_memory = await run_retrieve(client, query1)
+
+ # Step 4: Run agent with memory-augmented query
+ print("Running memory-augmented agent")
+ augmented_query = f"{retrieved_memory}\n\nUser Question:\n{query1}"
+ final_messages = await run_agent(client, query=augmented_query)
+
+ # Extract the agent's final answer
+ final_answer = ""
+ for msg in final_messages:
+ if msg.get("role") == "assistant" and msg.get("content"):
+ final_answer = msg.get("content")
+ break
+
+ print(f"Memory-augmented response: {final_answer}")
+
+
+# Run the workflow
+if __name__ == "__main__":
+ asyncio.run(memory_augmented_workflow())
```
-
+### Managing Vector Store with MCP
-### 🔍 Using the Retriever Tool
-
-Intelligently search and retrieve the most relevant experiences from your workspace.
-
-**Tool Parameters:**
-
-- `query`: Search query string
-- `messages`: List of conversation messages (optional)
-- `top_k`: Number of top experiences to retrieve (default: 1)
-- `workspace_id`: Workspace identifier (default: "default")
-- `config`: Additional configuration parameters (optional)
-
-
-Python MCP Client Example
+You can also manage your vector store through MCP:
```python
-import asyncio
-from experiencemaker.service.mcp_client import MCPClient
-from experiencemaker.schema.request import RetrieverRequest
+async def manage_vector_store(client):
+ # Delete a workspace
+ await client.call_tool(
+ "vector_store",
+ arguments={
+ "workspace_id": WORKSPACE_ID,
+ "action": "delete",
+ }
+ )
+ # Dump memories to disk
+ await client.call_tool(
+ "vector_store",
+ arguments={
+ "workspace_id": WORKSPACE_ID,
+ "action": "dump",
+ "path": "./backups/",
+ }
+ )
-async def example_retriever():
- async with MCPClient(base_url="http://0.0.0.0:8001/sse") as client:
- request = RetrieverRequest(
- workspace_id="math_workspace",
- query="How to solve basic arithmetic problems?",
- top_k=3
- )
-
- response = await client.call_retriever(request)
- print(f"Retrieved experiences: {response.experience_merged}")
- print(f"Experience list:")
- for exp in response.experience_list:
- print(f"- {exp.content}")
-
-
-# Run the example
-asyncio.run(example_retriever())
+ # Load memories from disk
+ await client.call_tool(
+ "vector_store",
+ arguments={
+ "workspace_id": WORKSPACE_ID,
+ "action": "load",
+ "path": "./backups/",
+ }
+ )
```
-
-
-
-MCP Tool Call (JSON)
-
-```json
-{
- "method": "tools/call",
- "params": {
- "name": "retriever",
- "arguments": {
- "query": "How to solve basic arithmetic problems?",
- "top_k": 3,
- "workspace_id": "math_workspace"
- }
- }
-}
-```
-
-
-
-### 💾 Using the Vector Store Tool
-
-Manage vector store operations for workspace data.
-
-**Tool Parameters:**
-
-- `action`: Action to perform ("dump", "load", "delete", "copy")
-- `workspace_id`: Target workspace identifier
-- `src_workspace_id`: Source workspace (for copy operation)
-- `path`: File system path (for dump/load operations, default: "./")
-- `config`: Additional configuration parameters (optional)
-
-#### Dump Experiences From Vector Store
-
-
-Python MCP Client Example
-
-```python
-import asyncio
-from experiencemaker.service.mcp_client import MCPClient
-from experiencemaker.schema.request import VectorStoreRequest
-
-
-async def example_dump():
- async with MCPClient(base_url="http://0.0.0.0:8001/sse") as client:
- request = VectorStoreRequest(
- workspace_id="math_workspace",
- action="dump",
- path="./backups/"
- )
-
- response = await client.call_vector_store(request)
- print(f"Dump result: {response}")
-
-
-# Run the example
-asyncio.run(example_dump())
-```
-
-
-
-
-MCP Tool Call (JSON)
-
-```json
-{
- "method": "tools/call",
- "params": {
- "name": "vector_store",
- "arguments": {
- "action": "dump",
- "workspace_id": "math_workspace",
- "path": "./backups/"
- }
- }
-}
-```
-
-
-
-#### Load Experiences To Vector Store
-
-
-Python MCP Client Example
-
-```python
-import asyncio
-from experiencemaker.service.mcp_client import MCPClient
-from experiencemaker.schema.request import VectorStoreRequest
-
-
-async def example_load():
- async with MCPClient(base_url="http://0.0.0.0:8001/sse") as client:
- request = VectorStoreRequest(
- workspace_id="math_workspace",
- action="load",
- path="./backups/"
- )
-
- response = await client.call_vector_store(request)
- print(f"Load result: {response}")
-
-
-# Run the example
-asyncio.run(example_load())
-```
-
-
-
-#### Delete Workspace
-
-
-Python MCP Client Example
-
-```python
-import asyncio
-from experiencemaker.service.mcp_client import MCPClient
-from experiencemaker.schema.request import VectorStoreRequest
-
-
-async def example_delete():
- async with MCPClient(base_url="http://0.0.0.0:8001/sse") as client:
- request = VectorStoreRequest(
- workspace_id="math_workspace",
- action="delete"
- )
-
- response = await client.call_vector_store(request)
- print(f"Delete result: {response}")
-
-
-# Run the example
-asyncio.run(example_delete())
-```
-
-
-
-#### Copy Workspace
-
-
-Python MCP Client Example
-
-```python
-import asyncio
-from experiencemaker.service.mcp_client import MCPClient
-from experiencemaker.schema.request import VectorStoreRequest
-
-
-async def example_copy():
- async with MCPClient(base_url="http://0.0.0.0:8001/sse") as client:
- request = VectorStoreRequest(
- workspace_id="math_workspace_copy",
- action="copy",
- src_workspace_id="math_workspace"
- )
-
- response = await client.call_vector_store(request)
- print(f"Copy result: {response}")
-
-
-# Run the example
-asyncio.run(example_copy())
-```
-
-
-
-## 🔄 Complete MCP Workflow Example
-
-Here's a complete example showing the full workflow:
-
-```python
-import asyncio
-from experiencemaker.service.mcp_client import MCPClient
-from experiencemaker.schema.request import SummarizerRequest, RetrieverRequest
-from experiencemaker.schema.message import Message, Trajectory, Role
-
-async def complete_workflow():
- async with MCPClient(base_url="http://0.0.0.0:8001/sse") as client:
- print("Available tools:", await client.list_tools())
-
- # Step 1: Create experiences from trajectories
- trajectory = Trajectory(
- messages=[
- Message(role=Role.USER, content="How do I calculate compound interest?"),
- Message(role=Role.ASSISTANT,
- content="Compound interest is calculated using the formula A = P(1 + r/n)^(nt), where A is the final amount, P is the principal, r is the annual interest rate, n is the number of times interest is compounded per year, and t is the time in years."),
- Message(role=Role.USER, content="Can you give me an example?"),
- Message(role=Role.ASSISTANT,
- content="Sure! If you invest $1000 at 5% annual interest compounded monthly for 2 years: A = 1000(1 + 0.05/12)^(12*2) = $1104.94")
- ],
- score=1.0
- )
-
- summarizer_request = SummarizerRequest(
- workspace_id="finance_workspace",
- traj_list=[trajectory]
- )
-
- summarizer_response = await client.call_summarizer(summarizer_request)
- print(f"Created {len(summarizer_response.experience_list)} experiences")
-
- # Step 2: Retrieve relevant experiences
- retriever_request = RetrieverRequest(
- workspace_id="finance_workspace",
- query="How to calculate interest on investments?",
- top_k=2
- )
-
- retriever_response = await client.call_retriever(retriever_request)
- print(f"Retrieved experiences: {retriever_response.experience_merged}")
-
-
-# Run the complete workflow
-asyncio.run(complete_workflow())
-```
-
-## 🎭 Claude Desktop Integration
-
-Once configured with Claude Desktop, you can directly ask Claude to use ExperienceMaker tools:
-
-```
-Claude, please use the summarizer tool to create experiences from this conversation about solving math problems, then retrieve similar experiences when I ask about arithmetic.
-```
-
-Claude will automatically call the appropriate MCP tools and provide contextually relevant responses based on your
-stored experiences.
-
-## 🐛 Common Issues
+## 🐛 Common Issues and Troubleshooting
### MCP Server Won't Start
-
- Check if the required ports are available (for SSE transport)
- Verify your API keys in `.env` file
- Ensure Python version is 3.12+
- Check MCP transport configuration
### MCP Client Connection Issues
-
- For STDIO: Ensure the command path is correct in your MCP client config
- For SSE: Verify the server URL and port accessibility
- Check firewall settings for SSE connections
-### No Experiences Retrieved
+### No Memories Retrieved
-- Make sure you've run the summarizer tool first to create experiences
+- Make sure you've run the summarizer tool first to create memories
- Check if workspace_id matches between operations
- Verify vector store backend is properly configured
### API Connection Errors
-
- Confirm LLM_BASE_URL and API keys are correct
- Test API access independently
- Check network connectivity
-
-## 🔧 Advanced Configuration
-
-### Custom MCP Client Setup
-
-```python
-# For STDIO transport
-async with MCPClient(enable_sse=False) as client:
- # Your MCP operations here
- pass
-
-# For SSE transport with custom URL
-async with MCPClient(base_url="http://custom-host:8001/sse") as client:
- # Your MCP operations here
- pass
-```
-
-### Server Configuration Options
-
-```bash
-# Full configuration example
-experiencemaker_mcp \
- mcp_transport=stdio \
- http_service.host=0.0.0.0 \
- http_service.port=8001 \
- llm.default.model_name=qwen3-32b \
- llm.default.api_key=${LLM_API_KEY} \
- llm.default.base_url=${LLM_BASE_URL} \
- embedding_model.default.model_name=text-embedding-v4 \
- embedding_model.default.api_key=${EMBEDDING_MODEL_API_KEY} \
- embedding_model.default.base_url=${EMBEDDING_MODEL_BASE_URL} \
- vector_store.default.backend=elasticsearch \
- vector_store.default.host=localhost \
- vector_store.default.port=9200
-```
-
----
-
-🎯 **You're all set!** You now have a working ExperienceMaker MCP setup that can seamlessly integrate with MCP-compatible
-clients and learn from interactions to improve over time through the standardized MCP protocol.
-
-## 📚 Next Steps
-
-- Explore the [Configuration Guide](configuration_guide.md) for advanced customization
-- Check out [cookbook examples](../cookbook/) for practical implementations
-- Learn about [Vector Store Setup](vector_store_setup.md) for production deployments
-- Review the [Operations Documentation](operations_documentation.md) for maintenance procedures
\ No newline at end of file
diff --git a/doc/operations_documentation.md b/doc/operations_documentation.md
deleted file mode 100644
index 58a420ad..00000000
--- a/doc/operations_documentation.md
+++ /dev/null
@@ -1,25 +0,0 @@
-# Operations Documentation
-
-This document provides an overview of all operations in the ExperienceMaker framework.
-
-## Operations Overview
-
-| Op Class | Registered Backend | Description | Parameters |
-|-----------------------------|-------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `BuildQueryOp` | `build_query_op` | Constructs retrieval queries from user requests. If request.query exists, uses it directly. If only messages are provided, can either use LLM-based query construction or create a simple summary from the last 3 messages (200 chars each). Sets search query and messages in context for downstream operations. | `op.build_query_op.params.enable_llm_build = true/false` - Enable LLM-based query construction from messages. When false, creates simple summary from last 3 messages |
-| `RerankExperienceOp` | `rerank_experience_op` | Performs two-stage experience reranking: (1) LLM-based intelligent reranking using relevance evaluation, (2) Score-based filtering using confidence and validation scores. Returns top-k results after filtering. Handles parsing of LLM reranking responses in JSON format with fallback to text parsing. | `op.rerank_experience_op.params.enable_llm_rerank = true` - Enable LLM-based reranking `op.rerank_experience_op.params.enable_score_filter = false` - Enable score-based filtering `op.rerank_experience_op.params.min_score_threshold = 0.3` - Minimum combined score threshold for filtering `op.rerank_experience_op.params.top_k = 5` - Number of top experiences to return after reranking |
-| `RewriteExperienceOp` | `rewrite_experience_op` | Intelligently rewrites experience context for better task relevance. Extracts current context from recent messages (last 3), formats experiences, and optionally uses LLM to rewrite context based on current query and conversation history. Handles JSON response parsing with fallback to original content. Generates structured context messages with "When to use" and "Content" sections. | `op.rewrite_experience_op.params.enable_llm_rewrite = true` - Enable LLM-based context rewriting to make experiences more relevant and actionable for current task |
-| `MergeExperienceOp` | `merge_experience_op` | Formats multiple experiences into a single structured context message. Creates "Previous Experience" header followed by bullet-pointed list of experiences with "when_to_use" and "content" fields. Adds guidance text encouraging comprehensive response using helpful parts from experiences. Simple concatenation-based approach without LLM processing. | No configurable parameters |
-| `TrajectoryPreprocessOp` | `trajectory_preprocess_op` | Validates and classifies trajectories based on success threshold scoring. Separates trajectories into success/failure categories and sets up context variables (success_trajectories, failure_trajectories, all_trajectories) for downstream extraction operations. Essential preprocessing step for all summarizer operations. | `op.trajectory_preprocess_op.params.success_threshold = 1.0` - Score threshold to classify trajectories as successful. Trajectories with scores >= threshold are classified as success |
-| `TrajectorySegmentationOp` | `trajectory_segmentation_op` | Uses LLM to segment trajectories into meaningful step sequences based on logical breakpoints. Supports selective segmentation of success, failure, or all trajectories. Parses LLM responses in JSON format with fallback to number extraction. Stores segmentation information in trajectory metadata for downstream operations. Formats trajectory content with step numbers and role information. | `op.trajectory_segmentation_op.params.segment_target = "all"` - Which trajectories to segment ("all", "success", "failure") |
-| `ExperienceValidationOp` | `experience_validation_op` | Validates extracted experiences using LLM-based quality assessment. Evaluates experiences for actionability, accuracy, relevance, clarity, and uniqueness. Uses parallel processing for efficiency. Parses JSON validation responses with score and validity flags. Filters experiences based on validation threshold and removes invalid ones with detailed logging of rejection reasons. | `op.experience_validation_op.params.validation_threshold = 0.5` - Minimum validation score threshold for experience acceptance. Experiences with scores below this threshold are filtered out |
-| `ExperienceDeduplicationOp` | `experience_deduplication_op` | Removes duplicate experiences using embedding-based similarity analysis. Compares against both existing vector store experiences and current batch experiences. Calculates cosine similarity between experience embeddings and filters duplicates above similarity threshold. Handles embedding generation failures gracefully and provides detailed logging of deduplication decisions. | `op.experience_deduplication_op.params.similarity_threshold = 0.5` - Cosine similarity threshold for duplicate detection `op.experience_deduplication_op.params.max_existing_experiences = 1000` - Maximum number of existing experiences to retrieve and compare against for deduplication |
-| `ComparativeExtractionOp` | `comparative_extraction_op` | Extracts insights by comparing different trajectory outcomes. Supports two comparison modes: (1) Soft comparison between highest and lowest scoring trajectories, (2) Hard comparison between similar success/failure step sequences using embedding-based similarity matching. Uses parallel processing and handles trajectory segmentation data when available. | `op.comparative_extraction_op.params.enable_soft_comparison = true` - Enable highest vs lowest score comparison `op.comparative_extraction_op.params.enable_similarity_comparison = false` - Enable success vs failure similarity comparison `op.comparative_extraction_op.params.max_similarity_sequences = 5` - Maximum sequences to compare for similarity `op.comparative_extraction_op.params.similarity_threshold = 0.3` - Similarity threshold for step sequence matching `op.comparative_extraction_op.params.max_similarity_pairs = 3` - Maximum similar pairs to extract experiences from |
-| `SimpleSummaryOp` | `simple_summary_op` | Generates basic experiences from individual trajectories using LLM-based analysis. Classifies trajectories as success/failure based on score threshold and creates structured experiences with when_to_use conditions and content. Parses JSON responses with robust error handling and validation. Uses parallel processing for multiple trajectories. | `op.simple_summary_op.params.success_score_threshold = 0.9` - Score threshold to classify trajectory as successful for experience extraction |
-| `SuccessExtractionOp` | `success_extraction_op` | Extracts actionable experiences from successful trajectories and their segments. Processes both segmented step sequences (when available) and entire trajectories. Uses parallel processing for efficiency. Merges message content and extracts trajectory context for rich experience generation. Creates TextExperience objects with proper metadata including workspace and author information. | No configurable parameters |
-| `FailureExtractionOp` | `failure_extraction_op` | Extracts learning experiences from failed trajectories to identify failure patterns and pitfalls. Similar to SuccessExtractionOp but focuses on failure analysis. Processes segmented sequences when available or entire trajectories. Uses parallel processing and creates structured experiences with proper metadata. Helps identify common failure modes and prevention strategies. | No configurable parameters |
-| `UpdateVectorStoreOp` | `update_vector_store_op` | Manages vector store updates through insert and delete operations. Handles deletion of experiences by ID list and insertion of new experience lists. Converts BaseExperience objects to VectorNode format for storage. Operates on workspace-specific vector databases with detailed logging of operation sizes and IDs. Supports batch operations for efficiency. | No configurable parameters - operations controlled by experience_list and deleted_experience_ids in response context |
-| `RecallVectorStoreOp` | `recall_vector_store_op` | Retrieves relevant experiences from vector store using semantic search. Performs content-based deduplication to avoid returning identical experiences. Supports optional score-based filtering to ensure quality results. Converts VectorNode results back to BaseExperience objects. Uses search query from context set by BuildQueryOp. | `op.recall_vector_store_op.params.threshold_score = ` - Optional minimum similarity score threshold for filtering search results. Results below this score are excluded |
-| `VectorStoreActionOp` | `vector_store_action_op` | Performs administrative operations on vector store workspaces. Supports four actions: (1) copy - duplicates workspace content, (2) delete - removes entire workspace, (3) dump - exports workspace to file with experience conversion, (4) load - imports workspace from file with node conversion. Handles callback functions for data transformation during dump/load operations. | Action-specific parameters: `request.action` ("copy"/"delete"/"dump"/"load"), `request.workspace_id` (target workspace), `request.src_workspace_id` (source workspace for copy), `request.path` (file path for dump/load) |
-| `ReactV1Op` | `react_v1_op` | Implements ReAct (Reasoning and Acting) agent framework for interactive problem-solving. Manages iterative reasoning-action cycles with configurable tools and step limits. Handles tool execution with parallel processing and result collection. Supports terminate tool for early stopping. Formats conversations with role prompts, tool responses, and final prompts. Includes built-in safeguards for missing tools and infinite loops. | `op.react_v1_op.params.max_steps = 10` - Maximum number of reasoning/action steps before termination `op.react_v1_op.params.tool_names = "code_tool,dashscope_search_tool,terminate_tool"` - Comma-separated list of available tools from the tool registry |
-
diff --git a/doc/personal_memory/personal_memory.md b/doc/personal_memory/personal_memory.md
new file mode 100644
index 00000000..1d2ac226
--- /dev/null
+++ b/doc/personal_memory/personal_memory.md
@@ -0,0 +1,124 @@
+# Personal Memory in Reme
+
+## Configuration Logic
+
+Reme's personal memory system consists of two main components: retrieval and summarization. The configuration for these components is defined in the default.yaml file.
+
+### Retrieval Configuration (`retrieve_personal_memory`)
+
+```yaml
+retrieve_personal_memory:
+ flow_content: set_query_op >> (extract_time_op | (retrieve_memory_op >> semantic_rank_op)) >> fuse_rerank_op
+```
+
+This flow performs the following operations:
+1. `set_query_op`: Prepares the query for memory retrieval
+2. Parallel paths:
+ - `extract_time_op`: Extracts time-related information from the query
+ - `retrieve_memory_op >> semantic_rank_op`: Retrieves memories and ranks them semantically
+3. `fuse_rerank_op`: Combines and reranks the results for final output
+
+### Summarization Configuration (`summary_personal_memory`)
+
+```yaml
+summary_personal_memory:
+ flow_content: info_filter_op >> (get_observation_op | get_observation_with_time_op | load_today_memory_op) >> contra_repeat_op >> update_vector_store_op
+```
+
+This flow performs the following operations:
+1. `info_filter_op`: Filters incoming information to extract relevant personal details
+2. Parallel paths for observation extraction:
+ - `get_observation_op`: Extracts general observations
+ - `get_observation_with_time_op`: Extracts observations with time context
+ - `load_today_memory_op`: Loads memories from the current day
+3. `contra_repeat_op`: Removes contradictions and repetitions
+4. `update_vector_store_op`: Stores the processed memories in the vector database
+
+## Basic Usage
+
+The following example demonstrates how to use personal memory in MemoryScope:
+
+### 1. Setup
+
+```python
+import asyncio
+import json
+import aiohttp
+
+# API base URL (default is http://0.0.0.0:8002)
+base_url = "http://0.0.0.0:8002"
+workspace_id = "personal_memory_demo"
+```
+
+### 2. Clear Existing Memories
+
+```python
+async with aiohttp.ClientSession() as session:
+ # Delete existing workspace memories
+ async with session.post(
+ f"{base_url}/vector_store",
+ json={
+ "action": "delete",
+ "workspace_id": workspace_id,
+ },
+ headers={"Content-Type": "application/json"}
+ ) as response:
+ result = await response.json()
+```
+
+### 3. Create Conversation with Personal Information
+
+```python
+# Example conversation with personal details
+messages = [
+ {"role": "user", "content": "My name is John Smith, I'm 28 years old"},
+ {"role": "assistant", "content": "Nice to meet you, John!"},
+ {"role": "user", "content": "I'm a software engineer working with Python"},
+ {"role": "assistant", "content": "I see, you're a Python engineer."},
+ # Additional conversation messages...
+]
+```
+
+### 4. Summarize Personal Memories
+
+```python
+async with session.post(
+ f"{base_url}/summary_personal_memory",
+ json={
+ "trajectories": [
+ {"messages": messages, "score": 1.0}
+ ],
+ "workspace_id": workspace_id,
+ },
+ headers={"Content-Type": "application/json"}
+) as response:
+ result = await response.json()
+```
+
+### 5. Retrieve Personal Memories
+
+```python
+# Example queries to retrieve personal information
+queries = [
+ "What's my name and age?",
+ "What do I do for work?",
+ "What are my hobbies?"
+]
+
+for query in queries:
+ async with session.post(
+ f"{base_url}/retrieve_personal_memory",
+ json={
+ "query": query,
+ "workspace_id": workspace_id,
+ },
+ headers={"Content-Type": "application/json"}
+ ) as response:
+ result = await response.json()
+ print(f"Query: {query}")
+ print(f"Answer: {result.get('answer', '')}")
+```
+
+## Complete Example
+
+For a complete working example, refer to `/cookbook/simple_demo/use_personal_memory_demo.py` in the Reme repository.
\ No newline at end of file
diff --git a/doc/personal_memory/personal_retrieve_ops.md b/doc/personal_memory/personal_retrieve_ops.md
new file mode 100644
index 00000000..dc2c7b5e
--- /dev/null
+++ b/doc/personal_memory/personal_retrieve_ops.md
@@ -0,0 +1,132 @@
+# Personal Memory Retrieve Ops
+
+## SetQueryOp
+
+### Functionality
+`SetQueryOp` prepares the query for memory retrieval by setting the query and its associated timestamp into the context. It's the first operation in the personal memory retrieval flow.
+
+### Parameters
+- `op.set_query_op.params.timestamp`: (Optional) Integer timestamp to use instead of the current time. If not provided, the current timestamp will be used.
+
+### Implementation Details
+The operation:
+1. Takes the query from the context (which is guaranteed to exist as a flow input requirement)
+2. Sets a timestamp (either current time or from parameters)
+3. Stores the query and timestamp as a tuple in the context for downstream operations
+
+## ExtractTimeOp
+
+### Functionality
+`ExtractTimeOp` identifies and extracts time-related information from the query. It uses an LLM to analyze the query text and determine any temporal references or constraints.
+
+### Parameters
+- `op.extract_time_op.params.language`: Language for time extraction (defaults to "en")
+
+### Implementation Details
+The operation:
+1. Checks if the query contains datetime keywords
+2. If time-related words are found, it prepares a prompt for the LLM with:
+ - System instructions
+ - Few-shot examples
+ - The user's query and current time
+3. Parses the LLM response to extract time information (year, month, day, etc.)
+4. Stores the extracted time dictionary in the context for downstream operations
+
+## RetrieveMemoryOp
+
+### Functionality
+`RetrieveMemoryOp` retrieves memories from the vector store based on the query. It extends the `RecallVectorStoreOp` class to provide memory retrieval functionality.
+
+### Parameters
+- `op.retrieve_memory_op.params.recall_key`: Key in the context to use as the query (default: "query")
+- `op.retrieve_memory_op.params.top_k`: Maximum number of memories to retrieve (default: 3)
+- `op.retrieve_memory_op.params.threshold_score`: (Optional) Minimum similarity score for memories (filters out memories below this threshold)
+
+### Implementation Details
+The operation:
+1. Retrieves the query from the context
+2. Searches the vector store for relevant memories based on the query
+3. Removes duplicate memories
+4. Filters memories by threshold score if specified
+5. Stores the retrieved memories in the context for downstream operations
+
+## SemanticRankOp
+
+### Functionality
+`SemanticRankOp` ranks memories based on their semantic relevance to the query using an LLM. This improves the quality of retrieved memories by considering deeper semantic relationships beyond vector similarity.
+
+### Parameters
+- `op.semantic_rank_op.params.enable_ranker`: Whether to enable semantic ranking (default: true)
+- `op.semantic_rank_op.params.output_memory_max_count`: Maximum number of memories to output (default: 10)
+
+### Implementation Details
+The operation:
+1. Retrieves the memory list from the context
+2. If ranking is enabled and there are more memories than the output limit:
+ - Removes duplicates based on content
+ - Formats memories for LLM ranking
+ - Asks the LLM to rank memories by relevance on a scale of 0.0 to 1.0
+ - Parses the ranking results and applies scores to memories
+3. Sorts memories by score
+4. Stores the ranked memories in the context for downstream operations
+
+## FuseRerankOp
+
+### Functionality
+`FuseRerankOp` performs the final reranking of memories by combining multiple factors: semantic scores, memory types, and temporal relevance. It also formats the final output.
+
+### Parameters
+- `op.fuse_rerank_op.params.fuse_score_threshold`: Minimum score threshold for memories (default: 0.1)
+- `op.fuse_rerank_op.params.fuse_ratio_dict`: Dictionary of memory type to score multiplier ratios (default: {"conversation": 0.5, "observation": 1, "obs_customized": 1.2, "insight": 2.0})
+- `op.fuse_rerank_op.params.fuse_time_ratio`: Score multiplier for time-relevant memories (default: 2.0)
+- `op.fuse_rerank_op.params.output_memory_max_count`: Maximum number of memories to output (default: 5)
+
+### Implementation Details
+The operation:
+1. Retrieves extracted time information and memory list from the context
+2. For each memory:
+ - Checks if the memory score is above the threshold
+ - Applies a type-based adjustment factor based on the memory type
+ - Determines time relevance by matching memory time metadata with extracted time
+ - Calculates the final score by multiplying the original score by type and time factors
+3. Sorts memories by the reranked scores
+4. Selects the top-K memories based on the output limit
+5. Formats memories for output with timestamps if available
+6. Stores both the formatted output and the memory list in the context
+
+## PrintMemoryOp
+
+### Functionality
+`PrintMemoryOp` formats the retrieved memories for display to the user. It provides a clean, structured representation of the memory content.
+
+### Parameters
+No specific parameters for this operation.
+
+### Implementation Details
+The operation:
+1. Retrieves the memory list from the context
+2. Formats each memory with:
+ - Memory index
+ - When to use information
+ - Content
+ - Additional metadata (if available)
+3. Joins the formatted memories into a single string
+4. Stores the formatted string in the context as the response answer
+
+## ReadMessageOp
+
+### Functionality
+`ReadMessageOp` fetches unmemorized chat messages from the context. This is useful for retrieving recent conversations that haven't been processed into memories yet.
+
+### Parameters
+- `op.read_message_op.params.contextual_msg_max_count`: Maximum number of contextual messages to retrieve (default: 10)
+
+### Implementation Details
+The operation:
+1. Retrieves chat messages from the context
+2. Filters for messages that:
+ - Are not marked as memorized
+ - Contain the target name
+3. Flattens the messages into a single list
+4. Sorts messages by creation time if available
+5. Stores the filtered messages back in the context
diff --git a/doc/personal_memory/personal_summary_ops.md b/doc/personal_memory/personal_summary_ops.md
new file mode 100644
index 00000000..3b62f458
--- /dev/null
+++ b/doc/personal_memory/personal_summary_ops.md
@@ -0,0 +1,145 @@
+# Personal Memory Summary Ops
+
+## InfoFilterOp
+
+### Purpose
+Filters messages based on information content scores, retaining only those that include significant information about the user.
+
+### Parameters
+- `op.info_filter_op.params.preserved_scores`: Comma-separated string of scores to preserve (default: "2,3")
+- `op.info_filter_op.params.info_filter_msg_max_size`: Maximum size of messages to process (default: 200)
+
+### Description
+This operation analyzes messages to determine which ones contain valuable personal information. It uses an LLM to score each message on a scale of 0-3:
+- 0: No user information
+- 1: Hypothetical or fictional content
+- 2: General or time-sensitive information
+- 3: Clear, important information or explicitly requested records
+
+Only messages with scores specified in `preserved_scores` are retained. Messages are also filtered to exclude those already memorized and to only include messages from the user.
+
+## GetObservationOp
+
+### Purpose
+Extracts general observations about the user from messages that don't contain time-related information.
+
+### Parameters
+No specific parameters for this operation.
+
+### Description
+This operation processes messages that don't contain time-related keywords. It uses an LLM to extract meaningful observations about the user from these messages. Each observation includes:
+- Content: The actual observation text
+- Keywords: Tags that indicate when this observation might be relevant
+- Source message: The original message that led to this observation
+
+The operation creates `PersonalMemory` objects with observation type "personal_info" for each extracted observation.
+
+## GetObservationWithTimeOp
+
+### Purpose
+Extracts observations with time context from messages that contain time-related information.
+
+### Parameters
+No specific parameters for this operation.
+
+### Description
+This operation is the counterpart to `GetObservationOp` but focuses specifically on messages containing time-related keywords. It extracts observations while preserving the time context, which is important for memories related to schedules, appointments, or time-specific preferences.
+
+The operation creates `PersonalMemory` objects with observation type "personal_info_with_time" for each extracted observation, including the time information in the metadata.
+
+## LoadTodayMemoryOp
+
+### Purpose
+Loads memories created today from the vector store to prevent duplication and enable updating of recent memories.
+
+### Parameters
+- `op.load_today_memory_op.params.top_k`: Maximum number of memories to retrieve (default: 50)
+
+### Description
+This operation retrieves memories created on the current day using vector store search with date filtering. It converts vector nodes to memory objects and makes them available for deduplication in subsequent operations. This helps ensure that new observations don't create redundant memories for information already captured earlier in the day.
+
+## ContraRepeatOp
+
+### Purpose
+Identifies and removes contradictory or repetitive information from the collected memories.
+
+### Parameters
+- `op.contra_repeat_op.params.contra_repeat_max_count`: Maximum number of memories to process (default: 50)
+- `op.contra_repeat_op.params.enable_contra_repeat`: Whether to enable contradiction/repetition checking (default: true)
+
+### Description
+This operation analyzes the combined memories from previous operations (observation_memories, observation_memories_with_time, today_memories) to identify contradictions or redundancies. It uses an LLM to evaluate each memory and mark it as:
+- "Contradiction": Contradicts other memories
+- "Contained": Redundant as the information is already contained in other memories
+- "None": Unique and should be kept
+
+Memories marked as contradictory or contained are filtered out, and their IDs are tracked for deletion from the vector store.
+
+## LongContraRepeatOp
+
+### Purpose
+Performs more sophisticated contradiction and redundancy analysis for longer-term memory management.
+
+### Parameters
+- `op.long_contra_repeat_op.params.long_contra_repeat_max_count`: Maximum number of memories to process (default: 50)
+- `op.long_contra_repeat_op.params.enable_long_contra_repeat`: Whether to enable this operation (default: true)
+
+### Description
+This operation extends the basic contradiction analysis of `ContraRepeatOp` with the ability to resolve conflicts by modifying contradictory memories rather than simply removing them. It's particularly useful for managing long-term personal memories where information might evolve over time.
+
+For contradictory memories, it can either:
+- Modify the content to resolve the contradiction
+- Remove the memory if it's completely invalidated
+- Keep the most accurate/recent information
+
+## UpdateInsightOp
+
+### Purpose
+Updates existing insight values based on new observations.
+
+### Parameters
+- `op.update_insight_op.params.update_insight_threshold`: Minimum relevance score threshold (default: 0.3)
+- `op.update_insight_op.params.update_insight_max_count`: Maximum number of insights to update (default: 5)
+
+### Description
+This operation integrates new observations into existing insights about the user. It:
+1. Scores insight memories based on relevance to new observations
+2. Selects the top insights that meet the relevance threshold
+3. Updates each selected insight using an LLM to incorporate the new information
+4. Creates updated insight memories with the original ID but new content
+
+This helps maintain accurate and up-to-date insights as new information about the user becomes available.
+
+## GetReflectionSubjectOp
+
+### Purpose
+Generates reflection subjects (topics) from personal memories for insight extraction.
+
+### Parameters
+- `op.get_reflection_subject_op.params.reflect_obs_cnt_threshold`: Minimum number of memories required for reflection (default: 10)
+- `op.get_reflection_subject_op.params.reflect_num_questions`: Maximum number of new subjects to generate (default: 3)
+
+### Description
+This operation analyzes a collection of personal memories to identify potential topics for reflection and insight generation. It:
+1. Checks if there are sufficient memories for meaningful reflection
+2. Extracts existing insight subjects to avoid duplication
+3. Uses an LLM to generate new reflection subjects based on memory content
+4. Creates insight memory objects for these new subjects
+
+The generated subjects serve as focal points for organizing and synthesizing personal information about the user.
+
+## UpdateVectorStoreOp
+
+### Purpose
+Stores the processed memories in the vector database and removes deleted memories.
+
+### Parameters
+No specific parameters for this operation.
+
+### Description
+This operation is the final step in the personal memory summarization flow. It:
+1. Deletes memories that were marked for removal (contradictory or redundant)
+2. Inserts new or updated memories into the vector store
+3. Records the number of deleted and inserted memories
+
+This ensures that the vector store remains up-to-date with the latest processed memories.
diff --git a/doc/quick_start.md b/doc/quick_start.md
deleted file mode 100644
index c4a4906b..00000000
--- a/doc/quick_start.md
+++ /dev/null
@@ -1,559 +0,0 @@
-# ExperienceMaker Quick Start Guide
-This guide will help you get started with ExperienceMaker quickly using practical examples.
-
-## 🚀 What You'll Learn
-- How to set up ExperienceMaker service
-- Run an agent and generate experiences
-- Retrieve and apply experiences to new tasks
-- Build experience-enhanced agents
-
-## 📋 Prerequisites
-- Python 3.12+
-- LLM API access (OpenAI or compatible)
-- Embedding model API access
-
-## 🛠️ Installation
-
-### Option 1: Install from PyPI (Recommended)
-
-```bash
-pip install experiencemaker
-```
-
-### Option 2: Install from Source
-
-```bash
-git clone https://github.com/modelscope/ExperienceMaker.git
-cd ExperienceMaker
-pip install .
-```
-
-## ⚙️ Environment Setup
-Create a `.env` file in your project directory:
-
-```bash
-# Required: LLM API configuration
-LLM_API_KEY="sk-xxx"
-LLM_BASE_URL="https://xxx.com/v1"
-
-# Required: Embedding model configuration
-EMBEDDING_MODEL_API_KEY="sk-xxx"
-EMBEDDING_MODEL_BASE_URL="https://xxx.com/v1"
-
-# Optional: Elasticsearch configuration (if using Elasticsearch backend)
-
-```
-
-## 🚀 Start the Service
-For testing, use the `local_file` backend:
-```bash
-experiencemaker \
- http_service.port=8001 \
- llm.default.model_name=qwen3-32b \
- embedding_model.default.model_name=text-embedding-v4 \
- vector_store.default.backend=local_file
-```
-The service will start on `http://localhost:8001`
-
-### Elasticsearch Backend
-```bash
-experiencemaker \
- http_service.port=8001 \
- llm.default.model_name=qwen3-32b \
- embedding_model.default.model_name=text-embedding-v4 \
- vector_store.default.backend=elasticsearch
-```
-
-**Setup Elasticsearch:**
-```bash
-export ES_HOSTS="http://localhost:9200"
-# Quick setup using Elastic's official script
-curl -fsSL https://elastic.co/start-local | sh
-```
-
-📖 **Need Help?** Refer to [Vector Store Setup](../../doc/vector_store_setup.md) for comprehensive deployment guidance.
-
-## 📝 Your First ExperienceMaker Script
-
-Here's how to get started!
-Note the `workspace_id` serves as your experience storage namespace. Experiences in different workspaces remain
-completely isolated and cannot access each other.
-
-### 📊 Call Summarizer Examples
-
-Transform conversation trajectories into valuable experiences using batch summarization. Each trajectory contains:
-
-- **Message**: Complete conversation history between user and agent
-- **Score**: Performance rating (0-1 scale, where 0=failure, 1=success)
-
-The summarizer analyzes these trajectories to extract actionable insights and patterns for future interactions.
-
-
-Python
-
-```python
-import requests
-
-response = requests.post(url="http://0.0.0.0:8001/summarizer", json={
- "workspace_id": "test_workspace",
- "traj_list": [
- {"messages": [{"role": "user", "content": "hello world"}], "score": 1.0}
- ]
-})
-
-experience_list = response.json()["experience_list"]
-for experience in experience_list:
- print(experience)
-```
-
-
-
-
-curl
-
-```bash
-curl -X POST "http://0.0.0.0:8001/summarizer" \
- -H "Content-Type: application/json" \
- -d '{
- "workspace_id": "test_workspace",
- "traj_list": [
- {
- "messages": [{"role": "user", "content": "hello world"}],
- "score": 1.0
- }
- ]
- }'
-```
-
-
-
-
-Node.js
-
-```javascript
-const fetch = require('node-fetch');
-// or: import fetch from 'node-fetch';
-
-async function callSummarizer() {
- try {
- const response = await fetch('http://0.0.0.0:8001/summarizer', {
- method: 'POST',
- headers: {
- 'Content-Type': 'application/json',
- },
- body: JSON.stringify({
- workspace_id: "test_workspace",
- traj_list: [
- {
- messages: [{ role: "user", content: "hello world" }],
- score: 1.0
- }
- ]
- })
- });
-
- const data = await response.json();
- const experienceList = data.experience_list;
-
- experienceList.forEach(experience => {
- console.log(experience);
- });
- } catch (error) {
- console.error('Error:', error);
- }
-}
-
-callSummarizer();
-```
-
-
-
-### 🔍 Call Retriever Examples
-
-Intelligently search and retrieve the most relevant experiences from your workspace to enhance decision-making. The retriever:
-
-- **Finds** the top-k most similar experiences based on semantic similarity to your query
-- **Returns** pre-assembled context ready for immediate use, or raw experience data for custom processing
-- **Leverages** your workspace's accumulated knowledge to provide contextually relevant insights
-
-
-Python
-
-```python
-import requests
-
-response = requests.post(url="http://0.0.0.0:8001/retriever", json={
- "workspace_id": "test_workspace",
- "query": "what is the meaning of life?",
- "top_k": 1,
-})
-
-experience_merged: str = response.json()["experience_merged"]
-print(f"experience_merged={experience_merged}")
-```
-
-
-
-
-curl
-
-```bash
-curl -X POST "http://0.0.0.0:8001/retriever" \
- -H "Content-Type: application/json" \
- -d '{
- "workspace_id": "test_workspace",
- "query": "what is the meaning of life?",
- "top_k": 1
- }'
-```
-
-
-
-
-Node.js
-
-```javascript
-const fetch = require('node-fetch');
-// or: import fetch from 'node-fetch';
-
-async function callRetriever() {
- try {
- const response = await fetch('http://0.0.0.0:8001/retriever', {
- method: 'POST',
- headers: {
- 'Content-Type': 'application/json',
- },
- body: JSON.stringify({
- workspace_id: "test_workspace",
- query: "what is the meaning of life?",
- top_k: 1
- })
- });
-
- const data = await response.json();
- const experienceMerged = data.experience_merged;
-
- console.log(`experience_merged=${experienceMerged}`);
- } catch (error) {
- console.error('Error:', error);
- }
-}
-
-callRetriever();
-```
-
-
-
-### 💾 Dump Experiences From Vector Store
-
-Export and backup your valuable experience data for archival, analysis, or migration purposes. This operation:
-
-- **Extracts** all experiences from the specified workspace in the vector store
-- **Saves** them to a structured JSONL file at `{path}/{workspace_id}.jsonl`
-- **Preserves** complete experience metadata and embeddings for future restoration
-
-
-Python
-
-```python
-import requests
-
-response = requests.post(url="http://0.0.0.0:8001/vector_store", json={
- "workspace_id": "test_workspace",
- "action": "dump",
- "path": "./",
-})
-print(response.json())
-```
-
-
-
-
-curl
-
-```bash
-curl -X POST "http://0.0.0.0:8001/vector_store" \
- -H "Content-Type: application/json" \
- -d '{
- "workspace_id": "test_workspace",
- "action": "dump",
- "path": "./"
- }'
-```
-
-
-
-
-Node.js
-
-```javascript
-const fetch = require('node-fetch');
-// or: import fetch from 'node-fetch';
-
-async function dumpExperiences() {
- try {
- const response = await fetch('http://0.0.0.0:8001/vector_store', {
- method: 'POST',
- headers: {
- 'Content-Type': 'application/json',
- },
- body: JSON.stringify({
- workspace_id: "test_workspace",
- action: "dump",
- path: "./"
- })
- });
-
- const data = await response.json();
- console.log(data);
- } catch (error) {
- console.error('Error:', error);
- }
-}
-
-dumpExperiences();
-```
-
-
-
-### 📥 Load Experiences To Vector Store
-
-Import and restore previously exported experience data to populate your workspace with existing knowledge. This operation:
-
-- **Reads** experience data from the JSONL file located at `{path}/{workspace_id}.jsonl`
-- **Reconstructs** the vector embeddings and indexes them in the specified workspace
-- **Enables** immediate access to imported experiences for retrieval and decision-making
-
-
-Python
-
-```python
-import requests
-
-response = requests.post(url="http://0.0.0.0:8001/vector_store", json={
- "workspace_id": "test_workspace",
- "action": "load",
- "path": "./",
-})
-
-print(response.json())
-```
-
-
-
-
-curl
-
-```bash
-curl -X POST "http://0.0.0.0:8001/vector_store" \
- -H "Content-Type: application/json" \
- -d '{
- "workspace_id": "test_workspace",
- "action": "load",
- "path": "./"
- }'
-```
-
-
-
-
-Node.js
-
-```javascript
-const fetch = require('node-fetch');
-// or: import fetch from 'node-fetch';
-
-async function loadExperiences() {
- try {
- const response = await fetch('http://0.0.0.0:8001/vector_store', {
- method: 'POST',
- headers: {
- 'Content-Type': 'application/json',
- },
- body: JSON.stringify({
- workspace_id: "test_workspace",
- action: "load",
- path: "./"
- })
- });
-
- const data = await response.json();
- console.log(data);
- } catch (error) {
- console.error('Error:', error);
- }
-}
-
-loadExperiences();
-```
-
-
-
-
-### 🗑️ Delete Workspace
-
-Permanently remove a workspace and all its associated experience data when it's no longer needed. This operation:
-
-- **Removes** all experiences, embeddings, and metadata from the specified workspace
-- **Frees up** storage space and computational resources
-- **Cannot be undone** - ensure you've backed up important data before deletion
-
-
-Python
-
-```python
-import requests
-
-response = requests.post(url="http://0.0.0.0:8001/vector_store", json={
- "workspace_id": "test_workspace",
- "action": "delete"
-})
-
-print(response.json())
-```
-
-
-
-
-curl
-
-```bash
-curl -X POST "http://0.0.0.0:8001/vector_store" \
- -H "Content-Type: application/json" \
- -d '{
- "workspace_id": "test_workspace",
- "action": "delete"
- }'
-```
-
-
-
-
-Node.js
-
-```javascript
-const fetch = require('node-fetch');
-// or: import fetch from 'node-fetch';
-
-async function deleteWorkspace() {
- try {
- const response = await fetch('http://0.0.0.0:8001/vector_store', {
- method: 'POST',
- headers: {
- 'Content-Type': 'application/json',
- },
- body: JSON.stringify({
- workspace_id: "test_workspace",
- action: "delete"
- })
- });
-
- const data = await response.json();
- console.log(data);
- } catch (error) {
- console.error('Error:', error);
- }
-}
-
-deleteWorkspace();
-```
-
-
-
-### 📋 Copy Workspace
-
-Duplicate an existing workspace to create a new one with identical experience data, perfect for experimentation or branching. This operation:
-
-- **Clones** all experiences and embeddings from the source workspace
-- **Creates** a new independent workspace with the copied data
-- **Preserves** original workspace while enabling safe testing and modifications in the copy
-
-
-Python
-
-```python
-import requests
-
-response = requests.post(url="http://0.0.0.0:8001/vector_store", json={
- "workspace_id": "test_workspace",
- "action": "copy",
- "src_workspace_id": "src_workspace"
-})
-
-print(response.json())
-```
-
-
-
-
-curl
-
-```bash
-curl -X POST "http://0.0.0.0:8001/vector_store" \
- -H "Content-Type: application/json" \
- -d '{
- "workspace_id": "test_workspace",
- "action": "copy",
- "src_workspace_id": "src_workspace"
- }'
-```
-
-
-
-
-Node.js
-
-```javascript
-const fetch = require('node-fetch');
-// or: import fetch from 'node-fetch';
-
-async function copyWorkspace() {
- try {
- const response = await fetch('http://0.0.0.0:8001/vector_store', {
- method: 'POST',
- headers: {
- 'Content-Type': 'application/json',
- },
- body: JSON.stringify({
- workspace_id: "test_workspace",
- action: "copy",
- src_workspace_id: "src_workspace"
- })
- });
-
- const data = await response.json();
- console.log(data);
- } catch (error) {
- console.error('Error:', error);
- }
-}
-
-copyWorkspace();
-```
-
-
-
-🎭 **Want to See It in Action?** We've prepared a [simple react agent](../../cookbook/simple_demo/simple_demo.py) that
-demonstrates how to enhance agent capabilities by integrating summarizer and retriever components, achieving
-significantly better performance.
-
-## 🐛 Common Issues
-
-### Service Won't Start
-- Check if port 8001 is available
-- Verify your API keys in `.env` file
-- Ensure Python version is 3.12+
-
-### No Experiences Retrieved
-- Make sure you've run the summarizer first
-- Check if workspace_id matches between operations
-- Verify vector store backend is properly configured
-
-### API Connection Errors
-- Confirm LLM_BASE_URL and API keys are correct
-- Test API access independently
-- Check network connectivity
-
----
-
-🎯 **You're all set!** You now have a working ExperienceMaker setup that can learn from interactions and improve over time.
\ No newline at end of file
diff --git a/doc/sop_memory/making_sop_memories.md b/doc/sop_memory/making_sop_memories.md
new file mode 100644
index 00000000..0970160a
--- /dev/null
+++ b/doc/sop_memory/making_sop_memories.md
@@ -0,0 +1,122 @@
+# SOP Memory: Combining Atomic Operations into Complex Workflows
+
+## 1. Background
+
+In LLM application development, we often need to combine multiple basic operations (atomic operations) into more complex
+workflows. These workflows can handle complex tasks such as data retrieval, code generation, multi-turn dialogues, and
+more. By combining these atomic operations into Standard Operating Procedures (SOPs), we can:
+
+- Improve code reusability
+- Simplify implementation of complex tasks
+- Standardize common workflows
+- Reduce development and maintenance costs
+
+This document introduces how to combine atomic operations (Ops) to form new composite operation tools using the FlowLLM
+framework.
+
+## 2. Technical Solution
+
+### 2.1 Atomic Operation Definition
+
+Each operation (Op) needs to define the following core attributes:
+
+```python
+class BaseOp:
+ description: str # Description of the operation
+ input_schema: Dict[str, ParamAttr] # Input parameter schema definition
+ output_schema: Dict[str, ParamAttr] # Output parameter schema definition
+```
+
+Where `ParamAttr` defines parameter type, whether it's required, and other attributes:
+
+```python
+class ParamAttr:
+ type: Type # Parameter type, such as str, int, Dict, etc.
+ required: bool = True # Whether it must be provided
+ default: Any = None # Default value
+ description: str = "" # Parameter description
+```
+
+### 2.2 SOP Composition Process
+
+#### Step 1: Create Atomic Operation Instances
+
+First, instantiate the required atomic operations:
+
+```python
+from flowllm.op.gallery.mock_op import MockOp
+from flowllm.op.search.tavily_search_op import TavilySearchOp
+from flowllm.op.agent.react_v2_op import ReactV2Op
+
+# Create atomic operation instances
+search_op = TavilySearchOp()
+react_op = ReactV2Op()
+summary_op = MockOp(
+ description="Summarize search results",
+ input_schema={"search_results": ParamAttr(type=str, description="Search results to summarize")},
+ output_schema={"summary": ParamAttr(type=str, description="Summarized content")}
+)
+```
+
+#### Step 2: Define Data Flow Between Operations
+
+Set up input-output relationships between operations, defining how data flows between them:
+
+```python
+# Set input parameter sources
+react_op.set_input("context",
+ "search_summary") # react_op's context parameter is retrieved from search_summary in memory
+
+# Set output parameter destinations
+search_op.set_output("results", "search_results") # search_op's results output to search_results in memory
+summary_op.set_output("summary", "search_summary") # summary_op's summary output to search_summary in memory
+```
+
+#### Step 3: Build Operation Flow Graph
+
+Use operators to build the operation flow graph, defining execution order and parallel relationships:
+
+```python
+# Build operation flow graph
+flow = search_op >> summary_op >> react_op
+
+# Or more complex flows
+# Parallel operations use the | operator, sequential operations use the >> operator
+complex_flow = (search_op >> summary_op) | (another_search_op >> another_summary_op) >> react_op
+```
+
+Operator explanation:
+
+- `>>`: Sequential execution, execute the next operation after the previous one completes
+- `|`: Parallel execution, execute multiple operations simultaneously
+
+#### Step 4: Create Composite Operation Class
+
+Encapsulate the built operation flow into a new composite operation class:
+
+```python
+
+class SearchAndReactOp(BaseToolOp):
+ description = "Search for information and generate a response based on search results"
+ input_schema = ...
+ output_schema = ...
+
+ def build_flow(self):
+ search_op = TavilySearchOp()
+ summary_op = MockOp()
+ react_op = ReactV2Op()
+
+ # Set data flow
+ search_op.set_output("results", "search_results")
+ summary_op.set_input("search_results", "search_results")
+ summary_op.set_output("summary", "search_summary")
+ react_op.set_input("context", "search_summary")
+ react_op.set_output("response", "response")
+
+ # Build operation flow graph
+ return search_op >> summary_op >> react_op
+
+ async def execute(self, inputs: Dict[str, Any]) -> Dict[str, Any]:
+ # Execute operation flow
+ return await self.flow.execute(inputs)
+```
\ No newline at end of file
diff --git a/doc/task_memory/task_memory.md b/doc/task_memory/task_memory.md
new file mode 100644
index 00000000..b00d0ef8
--- /dev/null
+++ b/doc/task_memory/task_memory.md
@@ -0,0 +1,202 @@
+# Task Memory in Reme
+
+Task Memory is a key component of Reme that allows AI agents to learn from past experiences and improve their performance on similar tasks in the future. This document explains how task memory works and how to use it in your applications.
+
+## What is Task Memory?
+
+Task Memory represents knowledge extracted from previous task executions, including:
+- Successful approaches to solving problems
+- Common pitfalls and failures to avoid
+- Comparative insights between different approaches
+
+Each task memory contains:
+- `when_to_use`: Conditions that indicate when this memory is relevant
+- `content`: The actual knowledge or experience to be applied
+- Metadata about the memory's source and utility
+
+## Configuration Logic
+
+Task Memory in Reme is configured through two main flows:
+
+### 1. Summary Task Memory
+
+The `summary_task_memory` flow processes conversation trajectories to extract meaningful memories:
+
+```yaml
+summary_task_memory:
+ flow_content: trajectory_preprocess_op >> (success_extraction_op|failure_extraction_op|comparative_extraction_op) >> memory_validation_op >> update_vector_store_op
+ description: "Summarizes conversation trajectories or messages into structured memory representations for long-term storage"
+```
+
+This flow:
+1. Preprocesses trajectories (`trajectory_preprocess_op`)
+2. Extracts memories based on success/failure/comparative analysis
+3. Validates memories (`memory_validation_op`)
+4. Updates the vector store (`update_vector_store_op`)
+
+A simplified version (`summary_task_memory_simple`) is also available for less complex use cases.
+
+### 2. Retrieve Task Memory
+
+The `retrieve_task_memory` flow fetches relevant memories based on a query:
+
+```yaml
+retrieve_task_memory:
+ flow_content: build_query_op >> recall_vector_store_op >> rerank_memory_op >> rewrite_memory_op
+ description: "Retrieves the most relevant top-k memory experiences from historical data based on the current query to enhance task-solving capabilities"
+```
+
+This flow:
+1. Builds a query from the input (`build_query_op`)
+2. Recalls relevant memories from the vector store (`recall_vector_store_op`)
+3. Reranks memories by relevance (`rerank_memory_op`)
+4. Rewrites memories for better context integration (`rewrite_memory_op`)
+
+A simplified version (`retrieve_task_memory_simple`) is also available.
+
+## Basic Usage
+
+Here's how to use Task Memory in your application:
+
+### Step 1: Set Up Your Environment
+
+```python
+import requests
+
+# API configuration
+BASE_URL = "http://0.0.0.0:8002/"
+WORKSPACE_ID = "your_workspace_id"
+```
+
+### Step 2: Run an Agent and Generate Memories
+
+```python
+# Run the agent with a query
+response = requests.post(
+ url=f"{BASE_URL}react",
+ json={"query": "Your query here"}
+)
+messages = response.json().get("messages", [])
+
+# Summarize the conversation to create task memories
+response = requests.post(
+ url=f"{BASE_URL}summary_task_memory",
+ json={
+ "workspace_id": WORKSPACE_ID,
+ "trajectories": [
+ {"messages": messages, "score": 1.0}
+ ]
+ }
+)
+```
+
+### Step 3: Retrieve Relevant Memories for a New Task
+
+```python
+# Retrieve memories relevant to a new query
+response = requests.post(
+ url=f"{BASE_URL}retrieve_task_memory",
+ json={
+ "workspace_id": WORKSPACE_ID,
+ "query": "Your new query here"
+ }
+)
+retrieved_memory = response.json().get("answer", "")
+```
+
+### Step 4: Use Retrieved Memories to Enhance Agent Performance
+
+```python
+# Augment a new query with retrieved memories
+augmented_query = f"{retrieved_memory}\n\nUser Question:\n{your_query}"
+
+# Run agent with the augmented query
+response = requests.post(
+ url=f"{BASE_URL}react",
+ json={"query": augmented_query}
+)
+```
+
+## Complete Example
+
+Here's a complete example workflow that demonstrates how to use task memory:
+
+```python
+def run_agent_with_memory(query_first, query_second):
+ # Run agent with second query to build initial memories
+ messages = run_agent(query=query_second)
+
+ # Summarize conversation to create memories
+ requests.post(
+ url=f"{BASE_URL}summary_task_memory",
+ json={
+ "workspace_id": WORKSPACE_ID,
+ "trajectories": [
+ {"messages": messages, "score": 1.0}
+ ]
+ }
+ )
+
+ # Retrieve relevant memories for the first query
+ response = requests.post(
+ url=f"{BASE_URL}retrieve_task_memory",
+ json={
+ "workspace_id": WORKSPACE_ID,
+ "query": query_first
+ }
+ )
+ retrieved_memory = response.json().get("answer", "")
+
+ # Run agent with first query augmented with retrieved memories
+ augmented_query = f"{retrieved_memory}\n\nUser Question:\n{query_first}"
+ return run_agent(query=augmented_query)
+```
+
+## Managing Task Memories
+
+### Delete a Workspace
+
+```python
+response = requests.post(
+ url=f"{BASE_URL}vector_store",
+ json={
+ "workspace_id": WORKSPACE_ID,
+ "action": "delete"
+ }
+)
+```
+
+### Dump Memories to Disk
+
+```python
+response = requests.post(
+ url=f"{BASE_URL}vector_store",
+ json={
+ "workspace_id": WORKSPACE_ID,
+ "action": "dump",
+ "path": "./"
+ }
+)
+```
+
+### Load Memories from Disk
+
+```python
+response = requests.post(
+ url=f"{BASE_URL}vector_store",
+ json={
+ "workspace_id": WORKSPACE_ID,
+ "action": "load",
+ "path": "./"
+ }
+)
+```
+
+## Advanced Features
+
+Reme also provides additional task memory operations:
+
+- `record_task_memory`: Update frequency and utility attributes of retrieved memories
+- `delete_task_memory`: Delete memories based on utility/frequency thresholds
+
+For more detailed examples, see the `use_task_memory_demo.py` file in the cookbook directory of the Reme project.
diff --git a/doc/task_memory/task_retrieve_ops.md b/doc/task_memory/task_retrieve_ops.md
new file mode 100644
index 00000000..a12e8698
--- /dev/null
+++ b/doc/task_memory/task_retrieve_ops.md
@@ -0,0 +1,73 @@
+# Task Memory Retrieval Operations
+
+## BuildQueryOp
+
+### Purpose
+
+Constructs a query for memory retrieval either from a direct query input or by analyzing conversation messages.
+
+### Functionality
+
+- If a direct `query` is provided in the context, it uses that query
+- If `messages` are provided in the context, it can:
+ - Use an LLM to generate a query based on the conversation context
+ - Or create a simple query from recent messages without using an LLM
+
+### Parameters
+
+- `op.build_query_op.params.enable_llm_build` (boolean, default: `true`):
+ - When `true`, uses an LLM to generate a query from conversation messages
+ - When `false`, creates a simple query by concatenating recent messages
+
+## RerankMemoryOp
+
+### Purpose
+
+Reranks and filters recalled memories to ensure the most relevant memories are prioritized.
+
+### Functionality
+
+- Reranks memories using LLM-based analysis (optional)
+- Filters memories based on quality scores (optional)
+- Returns the top-k most relevant memories
+
+### Parameters
+
+- `op.rerank_memory_op.params.enable_llm_rerank` (boolean, default: `true`):
+ - When `true`, uses an LLM to rerank memories based on their relevance to the query
+- `op.rerank_memory_op.params.enable_score_filter` (boolean, default: `false`):
+ - When `true`, filters memories based on their quality scores
+- `op.rerank_memory_op.params.min_score_threshold` (float, default: `0.3`):
+ - Minimum score threshold for filtering memories when `enable_score_filter` is `true`
+- `op.rerank_memory_op.params.top_k` (integer, default: `5`):
+ - Number of top memories to retain after reranking
+
+## RewriteMemoryOp
+
+### Purpose
+
+Rewrites and formats the retrieved memories to make them more relevant and actionable for the current context.
+
+### Functionality
+
+- Formats retrieved memories into a structured format
+- Can use an LLM to rewrite memories to better fit the current context (optional)
+- Generates a cohesive context message from multiple memories
+
+### Parameters
+
+- `op.rewrite_memory_op.params.enable_llm_rewrite` (boolean, default: `true`):
+ - When `true`, uses an LLM to rewrite the memories to make them more relevant and actionable
+ - When `false`, simply formats the memories without LLM-based rewriting
+
+## MergeMemoryOp
+
+### Purpose
+
+An alternative to RewriteMemoryOp that merges multiple memories into a single response without using an LLM.
+
+### Functionality
+
+- Collects the content from all memories in the memory list
+- Formats them into a single response with a standard structure
+- Adds a prompt to consider the helpful parts when answering the question
diff --git a/doc/task_memory/task_summary_ops.md b/doc/task_memory/task_summary_ops.md
new file mode 100644
index 00000000..d7c391aa
--- /dev/null
+++ b/doc/task_memory/task_summary_ops.md
@@ -0,0 +1,190 @@
+# Task Summary Operations
+
+## TrajectoryPreprocessOp
+
+### Purpose
+
+Preprocesses trajectories by validating and classifying them based on their score.
+
+### Functionality
+
+- Validates and classifies trajectories as success or failure based on a threshold
+- Modifies tool calls in messages to ensure consistent format
+- Sets context for downstream operators with classified trajectories
+
+### Parameters
+
+- `op.trajectory_preprocess_op.params.success_threshold` (float, default: `1.0`):
+ - The threshold score that determines if a trajectory is considered successful
+ - Trajectories with scores greater than or equal to this value are classified as successful
+
+## TrajectorySegmentationOp
+
+### Purpose
+
+Segments trajectories into meaningful step sequences to enable more granular memory extraction.
+
+### Functionality
+
+- Uses LLM to identify logical break points in trajectories
+- Adds segmentation information to trajectory metadata
+- Enables more focused memory extraction from specific parts of conversations
+
+### Parameters
+
+- `op.trajectory_segmentation_op.params.segment_target` (string, default: `"all"`):
+ - Determines which trajectories to segment
+ - Options: `"all"`, `"success"`, `"failure"`
+
+## SuccessExtractionOp
+
+### Purpose
+
+Extracts task memories from successful trajectories.
+
+### Functionality
+
+- Processes successful trajectories to identify valuable experiences
+- Can work with both entire trajectories and segmented step sequences
+- Uses LLM to extract structured task memories with when-to-use conditions
+
+### Parameters
+
+No specific parameters beyond the LLM configuration.
+
+## FailureExtractionOp
+
+### Purpose
+
+Extracts task memories from failed trajectories to capture lessons learned from unsuccessful attempts.
+
+### Functionality
+
+- Processes failed trajectories to identify pitfalls and mistakes
+- Can work with both entire trajectories and segmented step sequences
+- Uses LLM to extract structured task memories with when-to-use conditions
+
+### Parameters
+
+No specific parameters beyond the LLM configuration.
+
+## ComparativeExtractionOp
+
+### Purpose
+
+Extracts comparative task memories by comparing different scoring trajectories.
+
+### Functionality
+
+- Performs "soft comparison" between highest and lowest scoring trajectories
+- Can perform "hard comparison" between success and failure trajectories using similarity search
+- Identifies key differences that contributed to success or failure
+
+### Parameters
+
+- `op.comparative_extraction_op.params.enable_soft_comparison` (boolean, default: `true`):
+ - When `true`, enables comparison between highest and lowest scoring trajectories
+- `op.comparative_extraction_op.params.enable_similarity_comparison` (boolean, default: `false`):
+ - When `true`, enables similarity-based comparison between success and failure trajectories
+- `op.comparative_extraction_op.params.similarity_threshold` (float, default: `0.3`):
+ - The threshold for considering two trajectories similar
+- `op.comparative_extraction_op.params.max_similarity_sequences` (integer, default: `5`):
+ - Maximum number of sequences to compare to avoid computational overload
+- `op.comparative_extraction_op.params.max_similarity_pairs` (integer, default: `3`):
+ - Maximum number of similar pairs to process
+
+## MemoryValidationOp
+
+### Purpose
+
+Validates the quality of extracted task memories to ensure they are useful and relevant.
+
+### Functionality
+
+- Uses LLM to validate each extracted memory
+- Scores memories based on quality and relevance
+- Filters out low-quality memories based on validation threshold
+
+### Parameters
+
+- `op.memory_validation_op.params.validation_threshold` (float, default: `0.5`):
+ - The minimum score for a memory to be considered valid
+
+## MemoryDeduplicationOp
+
+### Purpose
+
+Removes duplicate task memories to avoid redundancy in the vector store.
+
+### Functionality
+
+- Compares new memories with existing memories in the vector store
+- Uses embedding similarity to identify duplicates
+- Ensures only unique memories are stored
+
+### Parameters
+
+- `op.memory_deduplication_op.params.similarity_threshold` (float, default: `0.5`):
+ - The threshold for considering two memories similar
+- `op.memory_deduplication_op.params.max_existing_task_memories` (integer, default: `1000`):
+ - Maximum number of existing memories to check against
+
+## SimpleSummaryOp
+
+### Purpose
+
+A simplified version of memory extraction that processes entire trajectories in one step.
+
+### Functionality
+
+- Classifies trajectories as success or failure based on score threshold
+- Extracts memories directly from complete trajectories
+- Useful for simpler use cases where detailed segmentation is not required
+
+### Parameters
+
+- `op.simple_summary_op.params.success_score_threshold` (float, default: `0.9`):
+ - The threshold score that determines if a trajectory is considered successful
+
+## SimpleComparativeSummaryOp
+
+### Purpose
+
+A simplified version of comparative memory extraction.
+
+### Functionality
+
+- Groups trajectories by task ID
+- Compares the highest and lowest scoring trajectories for each task
+- Extracts comparative insights without complex segmentation
+
+### Parameters
+
+No specific parameters beyond the LLM configuration.
+
+## PDFPreprocessOp
+
+### Purpose
+
+Processes PDF files to extract content that can be used for memory creation.
+
+### Functionality
+
+- Extracts text content from PDF files
+- Creates markdown representation of PDF content
+- Chunks content into manageable pieces for processing
+
+### Parameters
+
+- `op.pdf_preprocess_op.params.method` (string, default: `"auto"`):
+ - The method to use for PDF processing
+ - Options: `"auto"`, `"text"`, `"layout"`
+- `op.pdf_preprocess_op.params.lang` (string, default: `null` (auto-detect)):
+ - The language of the PDF content
+- `op.pdf_preprocess_op.params.backend` (string, default: `"pipeline"`):
+ - The backend to use for PDF processing
+ - Options: `"pipeline"`, `"pdfminer"`
+- `op.pdf_preprocess_op.params.create_chunks` (boolean, default: `true`):
+ - Whether to create chunks from the PDF content
+- `op.pdf_preprocess_op.params.max_chunk_length` (integer, default: `4000`):
+ - The maximum length of each chunk
\ No newline at end of file
diff --git a/doc/vector_store_api_guide.md b/doc/vector_store_api_guide.md
new file mode 100644
index 00000000..98f3eada
--- /dev/null
+++ b/doc/vector_store_api_guide.md
@@ -0,0 +1,390 @@
+# 🚀 Vector Store API Guide
+
+This guide covers the vector store implementations available in flowllm, their APIs, and how to use them effectively.
+
+## 📋 Overview
+
+flowllm provides multiple vector store backends for different use cases:
+
+- **LocalVectorStore** (`backend=local`) - 📁 Simple file-based storage for development and small datasets
+- **ChromaVectorStore** (`backend=chroma`) - 🔮 Embedded vector database for moderate scale
+- **EsVectorStore** (`backend=elasticsearch`) - 🔍 Elasticsearch-based storage for production and large scale
+
+All vector stores implement the `BaseVectorStore` interface, providing a consistent API across implementations.
+
+## 🔄 Common API Methods
+
+All vector store implementations share these core methods:
+
+### Workspace Management
+
+```python
+# Check if workspace exists
+store.exist_workspace(workspace_id: str) -> bool
+
+# Create a new workspace
+store.create_workspace(workspace_id: str, **kwargs)
+
+# Delete a workspace
+store.delete_workspace(workspace_id: str, **kwargs)
+
+# Copy a workspace
+store.copy_workspace(src_workspace_id: str, dest_workspace_id: str, **kwargs)
+```
+
+### Data Operations
+
+```python
+# Insert nodes (single or list)
+store.insert(nodes: VectorNode | List[VectorNode], workspace_id: str, **kwargs)
+
+# Delete nodes by ID
+store.delete(node_ids: str | List[str], workspace_id: str, **kwargs)
+
+# Search for similar nodes
+store.search(query: str, workspace_id: str, top_k: int = 1, **kwargs) -> List[VectorNode]
+
+# Iterate through workspace nodes
+for node in store.iter_workspace_nodes(workspace_id: str, **kwargs):
+ # Process each node
+```
+
+### Import/Export
+
+```python
+# Export workspace to file
+store.dump_workspace(workspace_id: str, path: str | Path = "", callback_fn=None, **kwargs)
+
+# Import workspace from file
+store.load_workspace(workspace_id: str, path: str | Path = "", nodes: List[VectorNode] = None,
+ callback_fn=None, **kwargs)
+```
+
+## ⚡ Vector Store Implementations
+
+### 1. 📁 LocalVectorStore (`backend=local`)
+
+A simple file-based vector store that saves data to local JSONL files.
+
+#### 💡 When to Use
+- **Development and testing** - No external dependencies required 🛠️
+- **Small datasets** - Suitable for datasets with < 10,000 vectors 📊
+- **Single-user applications** - Limited concurrent access support 👤
+
+#### ⚙️ Configuration
+
+```python
+from flowllm.storage.vector_store import LocalVectorStore
+from flowllm.embedding_model import OpenAICompatibleEmbeddingModel
+from flowllm.utils.common_utils import load_env
+
+# Load environment variables (for API keys)
+load_env()
+
+# Initialize embedding model
+embedding_model = OpenAICompatibleEmbeddingModel(dimensions=1024, model_name="text-embedding-v4")
+
+# Initialize vector store
+vector_store = LocalVectorStore(
+ embedding_model=embedding_model,
+ store_dir="./file_vector_store", # Directory to store JSONL files
+ batch_size=1024 # Batch size for operations
+)
+```
+
+#### 💻 Example Usage
+
+```python
+from flowllm.schema.vector_node import VectorNode
+
+# Create workspace
+workspace_id = "my_workspace"
+vector_store.create_workspace(workspace_id)
+
+# Create nodes
+nodes = [
+ VectorNode(
+ unique_id="node1",
+ workspace_id=workspace_id,
+ content="Artificial intelligence is revolutionizing technology",
+ metadata={"category": "tech", "source": "article1"}
+ ),
+ VectorNode(
+ unique_id="node2",
+ workspace_id=workspace_id,
+ content="Machine learning enables data-driven insights",
+ metadata={"category": "tech", "source": "article2"}
+ )
+]
+
+# Insert nodes
+vector_store.insert(nodes, workspace_id)
+
+# Search
+results = vector_store.search("What is AI?", workspace_id, top_k=2)
+for result in results:
+ print(f"Content: {result.content}")
+ print(f"Metadata: {result.metadata}")
+ print(f"Score: {result.metadata.get('score', 'N/A')}")
+```
+
+### 2. 🔮 ChromaVectorStore (`backend=chroma`)
+
+An embedded vector database that provides persistent storage with advanced features.
+
+#### 💡 When to Use
+- **Local development** with persistence requirements 🏠
+- **Medium-scale applications** (10K - 1M vectors) 📈
+- **Applications requiring metadata filtering** 🔍
+
+#### ⚙️ Configuration
+
+```python
+from flowllm.storage.vector_store import ChromaVectorStore
+from flowllm.embedding_model import OpenAICompatibleEmbeddingModel
+from flowllm.utils.common_utils import load_env
+
+# Load environment variables
+load_env()
+
+# Initialize embedding model
+embedding_model = OpenAICompatibleEmbeddingModel(dimensions=1024, model_name="text-embedding-v4")
+
+# Initialize vector store
+vector_store = ChromaVectorStore(
+ embedding_model=embedding_model,
+ store_dir="./chroma_vector_store", # Directory for Chroma database
+ batch_size=1024 # Batch size for operations
+)
+```
+
+#### 💻 Example Usage
+
+```python
+from flowllm.schema.vector_node import VectorNode
+
+workspace_id = "chroma_workspace"
+
+# Check if workspace exists and create if needed
+if not vector_store.exist_workspace(workspace_id):
+ vector_store.create_workspace(workspace_id)
+
+# Create nodes with metadata
+nodes = [
+ VectorNode(
+ unique_id="node1",
+ workspace_id=workspace_id,
+ content="Deep learning models require large datasets",
+ metadata={
+ "category": "AI",
+ "difficulty": "advanced",
+ "topic": "deep_learning"
+ }
+ ),
+ VectorNode(
+ unique_id="node2",
+ workspace_id=workspace_id,
+ content="Transformer architecture revolutionized NLP",
+ metadata={
+ "category": "AI",
+ "difficulty": "intermediate",
+ "topic": "transformers"
+ }
+ )
+]
+
+# Insert nodes
+vector_store.insert(nodes, workspace_id)
+
+# Search
+results = vector_store.search("deep learning", workspace_id, top_k=5)
+for result in results:
+ print(f"Content: {result.content}")
+ print(f"Metadata: {result.metadata}")
+```
+
+### 3. 🔍 EsVectorStore (`backend=elasticsearch`)
+
+Production-grade vector search using Elasticsearch with advanced filtering and scaling capabilities.
+
+#### 💡 When to Use
+- **Production environments** requiring high availability 🏭
+- **Large-scale applications** (1M+ vectors) 🚀
+- **Complex filtering requirements** on metadata 🎯
+
+#### 🛠️ Setup Elasticsearch
+
+Before using EsVectorStore, set up Elasticsearch:
+
+##### Option 1: Docker Run
+```bash
+# Pull the latest Elasticsearch image
+docker pull docker.elastic.co/elasticsearch/elasticsearch-wolfi:9.0.0
+
+# Run Elasticsearch container
+docker run -p 9200:9200 \
+ -e "discovery.type=single-node" \
+ -e "xpack.security.enabled=false" \
+ -e "xpack.license.self_generated.type=trial" \
+ -e "http.host=0.0.0.0" \
+ docker.elastic.co/elasticsearch/elasticsearch-wolfi:9.0.0
+```
+
+##### Environment Configuration
+```bash
+export FLOW_ES_HOSTS=http://localhost:9200
+```
+
+#### ⚙️ Configuration
+
+```python
+from flowllm.storage.vector_store import EsVectorStore
+from flowllm.embedding_model import OpenAICompatibleEmbeddingModel
+from flowllm.utils.common_utils import load_env
+import os
+
+# Load environment variables
+load_env()
+
+# Initialize embedding model
+embedding_model = OpenAICompatibleEmbeddingModel(dimensions=1024, model_name="text-embedding-v4")
+
+# Initialize vector store
+vector_store = EsVectorStore(
+ embedding_model=embedding_model,
+ hosts=os.getenv("FLOW_ES_HOSTS", "http://localhost:9200"), # Elasticsearch hosts
+ basic_auth=None, # ("username", "password") for auth
+ batch_size=1024 # Batch size for bulk operations
+)
+```
+
+#### 🎯 Advanced Filtering
+
+EsVectorStore supports advanced filtering capabilities:
+
+```python
+# Add term filters
+vector_store.add_term_filter("metadata.category", "technology")
+
+# Add range filters
+vector_store.add_range_filter("metadata.score", gte=0.8)
+vector_store.add_range_filter("metadata.timestamp", gte="2024-01-01", lte="2024-12-31")
+
+# Search with filters applied
+results = vector_store.search("machine learning", workspace_id, top_k=10)
+
+# Clear filters for next search
+vector_store.clear_filter()
+```
+
+#### 💻 Example Usage
+
+```python
+from flowllm.schema.vector_node import VectorNode
+
+# Define workspace
+workspace_id = "production_workspace"
+
+# Create workspace if needed
+if not vector_store.exist_workspace(workspace_id):
+ vector_store.create_workspace(workspace_id)
+
+# Create nodes with rich metadata
+nodes = [
+ VectorNode(
+ unique_id="doc1",
+ workspace_id=workspace_id,
+ content="Transformer architecture revolutionized NLP",
+ metadata={
+ "category": "AI",
+ "subcategory": "NLP",
+ "author": "research_team",
+ "timestamp": "2024-01-15",
+ "confidence": 0.95,
+ "tags": ["transformer", "nlp", "attention"]
+ }
+ )
+]
+
+# Insert with refresh for immediate availability
+vector_store.insert(nodes, workspace_id, refresh=True)
+
+# Advanced search with filters
+vector_store.add_term_filter("metadata.category", "AI")
+vector_store.add_range_filter("metadata.confidence", gte=0.9)
+
+results = vector_store.search("transformer models", workspace_id, top_k=5)
+
+for result in results:
+ print(f"Score: {result.metadata.get('score', 'N/A')}")
+ print(f"Content: {result.content}")
+ print(f"Metadata: {result.metadata}")
+```
+
+## 📝 Working with VectorNode
+
+The `VectorNode` class is the fundamental data unit for all vector stores:
+
+```python
+from flowllm.schema.vector_node import VectorNode
+
+# Create a node
+node = VectorNode(
+ unique_id="unique_identifier", # Unique ID for the node (required)
+ workspace_id="my_workspace", # Workspace ID (required)
+ content="Text content to embed", # Content to be embedded (required)
+ metadata={ # Optional metadata
+ "source": "document1",
+ "category": "technology",
+ "timestamp": "2024-08-29"
+ },
+ vector=None # Vector will be generated automatically if None
+)
+```
+
+## 🔄 Import/Export Example
+
+Export and import workspaces for backup or transfer:
+
+```python
+# Export workspace to file
+vector_store.dump_workspace(
+ workspace_id="my_workspace",
+ path="./backup_data" # Directory to store the exported data
+)
+
+# Import workspace from file
+vector_store.load_workspace(
+ workspace_id="new_workspace",
+ path="./backup_data" # Directory containing the exported data
+)
+
+# Copy workspace within the same store
+vector_store.copy_workspace(
+ src_workspace_id="original_workspace",
+ dest_workspace_id="copied_workspace"
+)
+```
+
+## 🧩 Integration with Embedding Models
+
+All vector stores require an embedding model to function:
+
+```python
+from flowllm.embedding_model import OpenAICompatibleEmbeddingModel
+
+# Initialize embedding model
+embedding_model = OpenAICompatibleEmbeddingModel(
+ dimensions=1024, # Embedding dimensions
+ model_name="text-embedding-v4", # Model name
+ batch_size=32 # Batch size for embedding generation
+)
+
+# Pass to vector store
+vector_store = LocalVectorStore(
+ embedding_model=embedding_model,
+ store_dir="./vector_store"
+)
+```
+
+🎉 This guide provides everything you need to work with vector stores in flowllm. Choose the implementation that best fits your use case and scale up as needed! ✨
\ No newline at end of file
diff --git a/doc/vector_store_setup.md b/doc/vector_store_setup.md
deleted file mode 100644
index a60b962c..00000000
--- a/doc/vector_store_setup.md
+++ /dev/null
@@ -1,257 +0,0 @@
-# 🚀 Vector Store Quick Start Guide
-This comprehensive guide covers all available vector store implementations in ExperienceMaker, their differences, use cases, and setup instructions.
-
-## 📋 Overview
-ExperienceMaker supports multiple vector store backends for different use cases and deployment scenarios:
-- **FileVectorStore** (`backend=local_file`) - 📁 Local file-based storage for development and small datasets
-- **ChromaVectorStore** (`backend=chroma`) - 🔮 Embedded vector database for local development and moderate scale
-- **EsVectorStore** (`backend=elasticsearch`) - 🔍 Elasticsearch-based storage for production and large scale
-
-## ⚡ Vector Store Implementations
-### 1. 📁 FileVectorStore (`backend=local_file`)
-A simple file-based vector store that saves data to local JSONL files. Perfect for development, testing, and small datasets.
-
-#### 💡 When to Use
-- **Development and testing** - No external dependencies required 🛠️
-- **Small datasets** - Suitable for datasets with < 10,000 vectors 📊
-- **Single-user applications** - No concurrent access support 👤
-- **Prototyping** - Quick setup without infrastructure ⚡
-
-#### ✨ Features
-- ✅ No external dependencies
-- ✅ Simple file-based persistence
-- ✅ Built-in cosine similarity search
-- ❌ No concurrent access support
-- ❌ Limited scalability
-- ❌ No advanced filtering
-
-#### ⚙️ Configuration Parameters
-```python
-from experiencemaker.vector_store import FileVectorStore
-from experiencemaker.embedding_model.openai_compatible_embedding_model import OpenAICompatibleEmbeddingModel
-
-embedding_model = OpenAICompatibleEmbeddingModel(dimensions=1024, model_name="text-embedding-v4")
-
-vector_store = FileVectorStore(
- embedding_model=embedding_model,
- store_dir="./file_vector_store", # Directory to store JSONL files
- batch_size=1024 # Batch size for operations
-)
-```
-
-#### 💻 Example Usage
-```python
-# Create workspace and insert data
-workspace_id = "my_workspace"
-vector_store.create_workspace(workspace_id)
-
-nodes = [
- VectorNode(
- workspace_id=workspace_id,
- content="Artificial intelligence is revolutionizing technology",
- metadata={"category": "tech", "source": "article1"}
- ),
- VectorNode(
- workspace_id=workspace_id,
- content="Machine learning enables data-driven insights",
- metadata={"category": "tech", "source": "article2"}
- )
-]
-
-vector_store.insert(nodes, workspace_id)
-
-# Search
-results = vector_store.search("What is AI?", workspace_id, top_k=2)
-```
-
-### 2. 🔮 ChromaVectorStore (`backend=chroma`)
-
-An embedded vector database that provides persistent storage with advanced features while remaining easy to deploy.
-
-#### 💡 When to Use
-- **Local development** with persistence requirements 🏠
-- **Medium-scale applications** (10K - 1M vectors) 📈
-- **Multi-user applications** with moderate concurrency 👥
-- **Applications requiring metadata filtering** 🔍
-- **Docker deployments** without external database dependencies 🐳
-
-#### ✨ Features
-- ✅ Persistent embedded database
-- ✅ Advanced metadata filtering
-- ✅ Built-in vector indexing (HNSW)
-- ✅ HTTP API support
-- ✅ Concurrent access support
-- ✅ Collection management
-- ❌ Limited horizontal scaling
-
-#### ⚙️ Configuration Parameters
-```python
-from experiencemaker.vector_store import ChromaVectorStore
-from experiencemaker.embedding_model.openai_compatible_embedding_model import OpenAICompatibleEmbeddingModel
-
-embedding_model = OpenAICompatibleEmbeddingModel(dimensions=1024, model_name="text-embedding-v4")
-
-vector_store = ChromaVectorStore(
- embedding_model=embedding_model,
- store_dir="./chroma_vector_store", # Directory for Chroma database
- batch_size=1024 # Batch size for operations
-)
-```
-
-#### 💻 Example Usage
-```python
-workspace_id = "chroma_workspace"
-
-# Check if workspace exists
-if not vector_store.exist_workspace(workspace_id):
- vector_store.create_workspace(workspace_id)
-
-# Insert with metadata
-nodes = [
- VectorNode(
- workspace_id=workspace_id,
- content="Deep learning models require large datasets",
- metadata={"category": "AI", "difficulty": "advanced", "topic": "deep_learning"}
- )
-]
-
-vector_store.insert(nodes, workspace_id)
-
-# Search with results
-results = vector_store.search("deep learning", workspace_id, top_k=5)
-for result in results:
- print(f"Content: {result.content}")
- print(f"Metadata: {result.metadata}")
-```
-
-### 3. 🔍 EsVectorStore (`backend=elasticsearch`)
-
-Production-grade vector search using Elasticsearch with advanced filtering, scaling, and enterprise features.
-
-#### 💡 When to Use
-- **Production environments** requiring high availability 🏭
-- **Large-scale applications** (1M+ vectors) 🚀
-- **High-throughput scenarios** with many concurrent users ⚡
-- **Complex filtering requirements** on metadata 🎯
-- **Distributed deployments** across multiple nodes 🌐
-- **Enterprise environments** with existing Elasticsearch infrastructure 🏢
-
-#### ✨ Features
-- ✅ Horizontal scaling
-- ✅ High availability and fault tolerance
-- ✅ Advanced filtering and aggregations
-- ✅ Real-time indexing and search
-- ✅ Cluster management
-- ✅ Enterprise security features
-- ✅ Monitoring and analytics
-- ❌ Complex setup and maintenance
-- ❌ Higher resource requirements
-
-#### 🛠️ Setup Elasticsearch
-
-Before using EsVectorStore, you need to set up Elasticsearch. Choose one of the following methods:
-
-##### Option 1: All-in-One Script (Recommended for Development) 🎯
-```bash
-curl -fsSL https://elastic.co/start-local | sh
-```
-
-##### Option 2: Docker Run with HTTP Host 🐳
-```bash
-# Pull the latest Elasticsearch image
-docker pull docker.elastic.co/elasticsearch/elasticsearch-wolfi:9.0.0
-
-# Run Elasticsearch container
-docker run -p 9200:9200 \
- -e "discovery.type=single-node" \
- -e "xpack.security.enabled=false" \
- -e "xpack.license.self_generated.type=trial" \
- -e "http.host=0.0.0.0" \
- docker.elastic.co/elasticsearch/elasticsearch-wolfi:9.0.0
-```
-
-##### 🔧 Environment Configuration
-Set the Elasticsearch hosts environment variable:
-```bash
-export ES_HOSTS=http://localhost:9200
-```
-
-#### ⚙️ Configuration Parameters
-```python
-from experiencemaker.vector_store import EsVectorStore
-from experiencemaker.embedding_model.openai_compatible_embedding_model import OpenAICompatibleEmbeddingModel
-import os
-
-embedding_model = OpenAICompatibleEmbeddingModel(dimensions=1024, model_name="text-embedding-v4")
-
-vector_store = EsVectorStore(
- embedding_model=embedding_model,
- hosts=os.getenv("ES_HOSTS", "http://localhost:9200"), # Elasticsearch hosts
- basic_auth=None, # ("username", "password") for auth
- batch_size=1024, # Batch size for bulk operations
- retrieve_filters=[] # Pre-configured filters
-)
-```
-
-#### 🎯 Advanced Filtering
-EsVectorStore supports advanced filtering capabilities:
-
-```python
-# Add term filters
-vector_store.add_term_filter("metadata.category", "technology")
-vector_store.add_term_filter("metadata.language", "en")
-
-# Add range filters
-vector_store.add_range_filter("metadata.score", gte=0.8)
-vector_store.add_range_filter("metadata.timestamp", gte="2024-01-01", lte="2024-12-31")
-
-# Search with filters applied
-results = vector_store.search("machine learning", workspace_id, top_k=10)
-
-# Clear filters for next search
-vector_store.clear_filter()
-```
-
-#### 💻 Example Usage
-```python
-from experiencemaker.schema.vector_node import VectorNode
-
-# Configure connection
-workspace_id = "production_workspace"
-
-# Create workspace with custom mapping
-if not vector_store.exist_workspace(workspace_id):
- vector_store.create_workspace(workspace_id)
-
-# Insert with rich metadata
-nodes = [
- VectorNode(
- workspace_id=workspace_id,
- content="Transformer architecture revolutionized NLP",
- metadata={
- "category": "AI",
- "subcategory": "NLP",
- "author": "research_team",
- "timestamp": "2024-01-15",
- "confidence": 0.95,
- "tags": ["transformer", "nlp", "attention"]
- }
- )
-]
-
-# Insert with refresh for immediate availability
-vector_store.insert(nodes, workspace_id, refresh=True)
-
-# Advanced search with filters
-vector_store.add_term_filter("metadata.category", "AI")
-vector_store.add_range_filter("metadata.confidence", gte=0.9)
-
-results = vector_store.search("transformer models", workspace_id, top_k=5)
-
-for result in results:
- print(f"Score: {result.metadata.get('_score', 'N/A')}")
- print(f"Content: {result.content}")
- print(f"Metadata: {result.metadata}")
-```
-
-🎉 This guide provides everything you need to get started with vector stores in ExperienceMaker. Choose the implementation that best fits your use case and scale up as needed! ✨
\ No newline at end of file
diff --git a/memoryscope/Dockerfile b/memoryscope/Dockerfile
deleted file mode 100644
index 68f4ab1d..00000000
--- a/memoryscope/Dockerfile
+++ /dev/null
@@ -1,55 +0,0 @@
-# __ __ ____
-# | \/ | ___ _ __ ___ ___ _ __ _ _/ ___| ___ ___ _ __ ___
-# | |\/| |/ _ \ '_ ` _ \ / _ \| '__| | | \___ \ / __/ _ \| '_ \ / _ \
-# | | | | __/ | | | | | (_) | | | |_| |___) | (_| (_) | |_) | __/
-# |_| |_|\___|_| |_| |_|\___/|_| \__, |____/ \___\___/| .__/ \___|
-# |___/ |_|
-
-# Instruction
-
-# To construct docker image:
-# sudo docker build --network=host -t memoryscope .
-
-# To run docker image:
-# sudo docker run -it --rm --memory=4G --net=host memoryscope
-# To run docker image with arguments (refer to memoryscope/core/config/arguments.py):
-# sudo docker run -it --rm --memory=4G --net=host -e "OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" -e "language=en" -e "human_name=superman" -e "generation_backend=openai_generation" -e "generation_model=gpt-4o" -e "embedding_backend=openai_embedding" -e "embedding_model=text-embedding-3-small" -e "enable_ranker=False" memoryscope
-
-FROM python:3.11
-
-# (Not necessary) Change pip source
-RUN echo '[global]' > /etc/pip.conf && \
- echo 'index-url = https://mirrors.aliyun.com/pypi/simple/' >> /etc/pip.conf && \
- echo 'trusted-host = mirrors.aliyun.com' >> /etc/pip.conf
-
-# Install Elastic Search
-RUN useradd -m elastic_search_user
-USER elastic_search_user
-WORKDIR /home/elastic_search_user/elastic_search
-# COPY elasticsearch-8.15.0-linux-x86_64.tar.gz ./elasticsearch-8.15.0-linux-x86_64.tar.gz
-RUN wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.15.0-linux-x86_64.tar.gz
-RUN tar -xzf elasticsearch-8.15.0-linux-x86_64.tar.gz
-WORKDIR /home/elastic_search_user/elastic_search/elasticsearch-8.15.0
-ENV DISCOVERY_TYPE=single-node \
- XPACK_SECURITY_ENABLED=false \
- XPACK_LICENSE_SELF_GENERATED_TYPE=trial
-
-# Change user back to root and fix ownership
-USER root
-RUN chown -R elastic_search_user:elastic_search_user /home/elastic_search_user/
-WORKDIR /memory_scope_project
-
-# (Not necessary) Install the majority of deps, using docker build cache to accelerate future building
-COPY requirements.txt ./
-RUN pip3 install -r requirements.txt
-
-# Enter working dir
-WORKDIR /memory_scope_project
-COPY . .
-# RUN pip3 install poetry
-# RUN poetry install
-RUN pip3 install -r requirements.txt
-
-# Launch!
-# CMD ["bash"]
-CMD ["bash", "examples/docker/entrypoint.sh"]
\ No newline at end of file
diff --git a/memoryscope/DockerfileArm b/memoryscope/DockerfileArm
deleted file mode 100644
index 6166f0fa..00000000
--- a/memoryscope/DockerfileArm
+++ /dev/null
@@ -1,56 +0,0 @@
-# __ __ ____
-# | \/ | ___ _ __ ___ ___ _ __ _ _/ ___| ___ ___ _ __ ___
-# | |\/| |/ _ \ '_ ` _ \ / _ \| '__| | | \___ \ / __/ _ \| '_ \ / _ \
-# | | | | __/ | | | | | (_) | | | |_| |___) | (_| (_) | |_) | __/
-# |_| |_|\___|_| |_| |_|\___/|_| \__, |____/ \___\___/| .__/ \___|
-# |___/ |_|
-
-# Instruction
-
-# To construct docker image:
-# sudo docker build --network=host -t memoryscope .
-
-# To run docker image:
-# sudo docker run -it --rm --memory=4G --net=host memoryscope
-# To run docker image with arguments (refer to memoryscope/core/config/arguments.py):
-# sudo docker run -it --rm --memory=4G --net=host -e "OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" -e "language=en" -e "human_name=superman" -e "generation_backend=openai_generation" -e "generation_model=gpt-4o" -e "embedding_backend=openai_embedding" -e "embedding_model=text-embedding-3-small" -e "enable_ranker=False" memoryscope
-#docker run -it --rm ghcr.io/modelscope/memoryscope_arm /bin/bash
-FROM python:3.11
-
-# (Not necessary) Change pip source
-RUN echo '[global]' > /etc/pip.conf && \
- echo 'index-url = https://mirrors.aliyun.com/pypi/simple/' >> /etc/pip.conf && \
- echo 'trusted-host = mirrors.aliyun.com' >> /etc/pip.conf
-
-# Install Elastic Search
-RUN useradd -m elastic_search_user
-USER elastic_search_user
-WORKDIR /home/elastic_search_user/elastic_search
-RUN wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.15.2-linux-aarch64.tar.gz
-RUN tar -xzf elasticsearch-8.15.2-linux-aarch64.tar.gz
-RUN mv /home/elastic_search_user/elastic_search/elasticsearch-8.15.2 /home/elastic_search_user/elastic_search/elasticsearch-8.15.0
-WORKDIR /home/elastic_search_user/elastic_search/elasticsearch-8.15.0
-ENV DISCOVERY_TYPE=single-node \
- XPACK_SECURITY_ENABLED=false \
- XPACK_LICENSE_SELF_GENERATED_TYPE=trial
-
-# Change user back to root and fix ownership
-USER root
-RUN chown -R elastic_search_user:elastic_search_user /home/elastic_search_user/
-WORKDIR /memory_scope_project
-
-# (Not necessary) Install the majority of deps, using docker build cache to accelerate future building
-COPY requirements.txt ./
-RUN pip3 install -r requirements.txt
-
-# Enter working dir
-WORKDIR /memory_scope_project
-COPY . .
-# RUN pip3 install poetry
-# RUN poetry install
-RUN pip3 install -r requirements.txt
-
-# Launch!
-# CMD ["bash"]
-CMD ["bash", "examples/docker/entrypoint.sh"]
-
diff --git a/memoryscope/README.md b/memoryscope/README.md
deleted file mode 100644
index f2e7143c..00000000
--- a/memoryscope/README.md
+++ /dev/null
@@ -1,131 +0,0 @@
-English | [**中文**](./README_ZH.md) | [**日本語**](./README_JP.md)
-
-# MemoryScope
-
-
-
-Equip your LLM chatbot with a powerful and flexible long term memory system.
-
-[](https://pypi.org/project/memoryscope/)
-[](https://pypi.org/project/memoryscope/)
-[](./LICENSE)
-[](https://modelscope.github.io/MemoryScope/en/index.html#welcome-to-memoryscope-tutorial)
-[](https://modelscope.github.io/MemoryScope/en/docs/api.html)
-[](https://modelscope.github.io/MemoryScope/en/docs/contribution.html)
-
-----
-## 📰 News
-
-- **[2024-09-10]** We release MemoryScope v0.1.1.0 now, which is also available in [PyPI](https://pypi.org/simple/memoryscope/)!
-----
-## 🌟 What is MemoryScope？
-MemoryScope provides LLM chatbots with powerful and flexible long-term memory capabilities, offering a framework for building such abilities.
-It can be applied to scenarios like personal assistants and emotional companions, continuously learning through long-term memory to remember users' basic information as well as various habits and preferences.
-This allows users to gradually experience a sense of "understanding" when using the LLM.
-
-### Demo
-
-
-
-
-### Framework
-
-
-
-
-💾 Memory Database: MemoryScope is equipped with a vector database (default is *ElasticSearch*) to store all memory fragments recorded in the system.
-
-🔧 Worker Library: MemoryScope atomizes the capabilities of long-term memory into individual workers, including over 20 workers for tasks such as query information filtering, observation extraction, and insight updating.
-
-🛠️ Operation Library: Based on the worker pipeline, it constructs the operations for memory services, realizing key capabilities such as memory retrieval and memory consolidation.
-
-- Memory Retrieval: Upon arrival of a user query, this operation returns the semantically related memory pieces
-and/or those from the corresponding time if the query involves reference to time.
-- Memory Consolidation: This operation takes in a batch of user queries and returns important user information
-extracted from the queries as consolidated *observations* to be stored in the memory database.
-- Reflection and Re-consolidation: At regular intervals, this operation performs reflection upon newly recorded *observations*
-to form and update *insights*. Then, memory re-consolidation is performed to ensure contradictions and repetitions
-among memory pieces are properly handled.
-
-
-⚙️ Best Practices:
-
-- Based on the core capabilities of long-term memory, MemoryScope has implemented a dialogue interface (API) with long-term memory and a command-line dialogue practice (CLI) with long-term memory.
-- MemoryScope combines currently popular agent frameworks (AutoGen, AgentScope) to provide best practices.
-
-### Main Features
-
-⚡ Low response-time (RT) for the user:
-- Backend operations (Memory Consolidation, Reflection and Re-consolidation) are decoupled from the frontend operation
- (Memory Retrieval) in the system.
-- While backend operations are usually (and are recommended to be) queued or executed at regular intervals, the
-system's response time (RT) for the user depends solely on the frontend operation, which is only ~500ms.
-
-🌲 Hierarchical and coherent memory:
-- The memory pieces stored in the system are in a hierarchical structure, with *insights* being the high level information
-from the aggregation of similarly-themed *observations*.
-- Contradictions and repetitions among memory pieces are handled periodically to ensure coherence of memory.
-- Fictitious contents from the user are filtered out to avoid hallucinations by the LLM.
-
-⏰ Time awareness:
-- The system is time sensitive when performing both Memory Retrieval and Memory Consolidation. Therefore, it can retrieve
-accurate relevant information when the query involves reference to time.
-
-----
-
-## 💼 Supported Model API
-
-| Backend | Task | Some Supported Models |
-|-------------------|------------|------------------------------------------------------------------------|
-| openai_backend | Generation | gpt-4o, gpt-4o-mini, gpt-4, gpt-3.5-turbo |
-| | Embedding | text-embedding-ada-002, text-embedding-3-large, text-embedding-3-small |
-| dashscope_backend | Generation | qwen-max, qwen-plus, qwen-plus, qwen2-72b-instruct |
-| | Embedding | text-embedding-v1, text-embedding-v2 |
-| | Reranker | gte-rerank |
-
-In the future, we will support more model interfaces and local deployment of LLM and embedding services.
-
-
-## 🚀 Installation
-For installation, please refer to [Installation.md](docs/installation.md).
-
-
-## 🍕 Quick Start
-- [Simple Usages (Quick Start)](./examples/api/simple_usages.ipynb)
-- [With AutoGen](./examples/api/autogen_example.md)
-- [CLI with a MemoryScope Chatbot](./examples/cli/CLI_README.md)
-- [Advanced Customization](./examples/advance/custom_operator.md)
-
-## 💡 Contribute
-
-Contributions are always encouraged!
-
-We highly recommend install pre-commit hooks in this repo before committing pull requests.
-These hooks are small house-keeping scripts executed every time you make a git commit,
-which will take care of the formatting and linting automatically.
-```shell
-pip install -e .
-pre-commit install
-```
-
-Please refer to our [Contribution Guide](./docs/contribution.md) for more details.
-
-## 📖 Citation
-
-Reference to cite if you use MemoryScope in a paper:
-
-```
-@software{MemoryScope,
-author = {Li Yu and
- Tiancheng Qin and
- Qingxu Fu and
- Sen Huang and
- Xianzhe Xu and
- Zhaoyang Liu and
- Boyin Liu},
-month = {09},
-title = {{MemoryScope}},
-url = {https://github.com/modelscope/MemoryScope},
-year = {2024}
-}
-```
diff --git a/memoryscope/README_JP.md b/memoryscope/README_JP.md
deleted file mode 100644
index 8186eb90..00000000
--- a/memoryscope/README_JP.md
+++ /dev/null
@@ -1,121 +0,0 @@
-[**English**](./README.md) | [**中文**](./README_ZH.md) | 日本語
-
-# MemoryScope
-
diff --git a/memoryscope/docs/sphinx_doc/en/source/_templates/layout.html b/memoryscope/docs/sphinx_doc/en/source/_templates/layout.html
deleted file mode 100644
index 1d182d30..00000000
--- a/memoryscope/docs/sphinx_doc/en/source/_templates/layout.html
+++ /dev/null
@@ -1,3 +0,0 @@
-
-{% extends "!layout.html" %} {% block sidebartitle %} {{ super() }} {% include
-"language_selector.html" %} {% endblock %}
diff --git a/memoryscope/docs/sphinx_doc/en/source/conf.py b/memoryscope/docs/sphinx_doc/en/source/conf.py
deleted file mode 100644
index ca707ba1..00000000
--- a/memoryscope/docs/sphinx_doc/en/source/conf.py
+++ /dev/null
@@ -1,105 +0,0 @@
-# -*- coding: utf-8 -*-
-# Configuration file for the Sphinx documentation builder.
-#
-# This file only contains a selection of the most common options. For a full
-# list see the documentation:
-# https://www.sphinx-doc.org/en/master/usage/configuration.html
-
-# -- Path setup --------------------------------------------------------------
-
-# If extensions (or modules to document with autodoc) are in another directory,
-# add these directories to sys.path here. If the directory is relative to the
-# documentation root, use os.path.abspath to make it absolute, like shown here.
-#
-import os
-import sys
-
-sys.path.insert(0, os.path.abspath("../../../../../MemoryScope"))
-
-
-# -- Project information -----------------------------------------------------
-
-language = "en"
-
-project = "MemoryScope"
-copyright = "2024, Alibaba Tongyi Lab"
-author = "EcoML team of Alibaba Tongyi Lab"
-
-
-# -- General configuration ---------------------------------------------------
-
-# Add any Sphinx extension module names here, as strings. They can be
-# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
-# ones.
-extensions = [
- "sphinx.ext.autodoc",
- "sphinx.ext.autosummary",
- "sphinx.ext.viewcode",
- "sphinx.ext.napoleon",
- "sphinxcontrib.mermaid",
- "myst_parser",
- "sphinx.ext.autosectionlabel",
- "sphinxcontrib.autodoc_pydantic",
- "nbsphinx"
-]
-
-autodoc_pydantic_model_show_json = True
-autodoc_pydantic_settings_show_json = True
-
-# Prefix document path to section labels, otherwise autogenerated labels would
-# look like 'heading' rather than 'path/to/file:heading'
-autosectionlabel_prefix_document = True
-autosummary_generate = True
-autosummary_ignore_module_all = False
-
-autodoc_member_order = "bysource"
-
-# If true, '()' will be appended to :func: etc. cross-reference text.
-add_function_parentheses = False
-
-# If true, the current module name will be prepended to all description
-# unit titles (such as .. function::).
-add_module_names = True
-
-autodoc_default_flags = ["members"]
-
-autodoc_default_options = {
- "members": True,
- "member-order": "bysource",
- "special-members": "__init__",
-}
-# Add any paths that contain templates here, relative to this directory.
-templates_path = ["_templates"]
-
-# List of patterns, relative to source directory, that match files and
-# directories to ignore when looking for source files.
-# This pattern also affects html_static_path and html_extra_path.
-exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
-
-# -- Options for HTML output -------------------------------------------------
-
-# The theme to use for HTML and HTML Help pages. See the documentation for
-# a list of builtin themes.
-#
-html_theme = "sphinx_rtd_theme"
-
-# html_logo = "_static/logo.png"
-
-# Add any paths that contain custom static files (such as style sheets) here,
-# relative to this directory. They are copied after the builtin static files,
-# so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ["_static"]
-
-html_theme_options = {
- # "logo_only": True,
- "navigation_depth": 4,
-}
-
-source_suffix = {
- ".rst": "restructuredtext",
- ".md": "markdown",
-}
-
-html_css_files = [
- "custom.css",
-]
diff --git a/memoryscope/docs/sphinx_doc/en/source/docs/api.rst b/memoryscope/docs/sphinx_doc/en/source/docs/api.rst
deleted file mode 100644
index e7ddd17f..00000000
--- a/memoryscope/docs/sphinx_doc/en/source/docs/api.rst
+++ /dev/null
@@ -1,68 +0,0 @@
-.. _api:
-
-
-MemoryScope API Documentation
-
-
-Enumeration
-===========
-
-.. automodule:: memoryscope.enumeration
- :members:
-
-Scheme
-======
-.. automodule:: memoryscope.scheme
- :members:
-
-Config
-======
-.. automodule:: memoryscope.core.config
- :members:
-
-
-Models
-======
-.. automodule:: memoryscope.core.models
- :members:
-
-
-
-Storage
-=======
-.. automodule:: memoryscope.core.storage
- :members:
-
-
-Worker
-======
-Base
-----
-
-.. automodule:: memoryscope.core.worker
- :members:
-
-Frontend
---------
-.. automodule:: memoryscope.core.worker.frontend
- :members:
-
-Backend
---------
-.. automodule:: memoryscope.core.worker.backend
- :members:
-
-Operation
-=========
-.. automodule:: memoryscope.core.operation
- :members:
-
-Service
-=======
-.. automodule:: memoryscope.core.service
- :members:
-
-Chat
-====
-.. automodule:: memoryscope.core.chat
- :members:
diff --git a/memoryscope/docs/sphinx_doc/en/source/index.rst b/memoryscope/docs/sphinx_doc/en/source/index.rst
deleted file mode 100644
index 825b4c4e..00000000
--- a/memoryscope/docs/sphinx_doc/en/source/index.rst
+++ /dev/null
@@ -1,57 +0,0 @@
-.. MemoryScope documentation master file, created by
- sphinx-quickstart on Fri Jan 5 17:53:54 2024.
- You can adapt this file completely to your liking, but it should at least
- contain the root `toctree` directive.
-
-:github_url: https://github.com/modelscope/memoryscope
-
-MemoryScope Documentation
-=========================
-
-Welcome to MemoryScope Tutorial
--------------------------------
-
-.. image:: docs/images/logo.png
- :align: center
-
-MemoryScope provides LLM chatbots with powerful and flexible long-term memory capabilities, offering a framework for building such abilities.
-It can be applied to scenarios like personal assistants and emotional companions, continuously learning through long-term memory to remember users' basic information as well as various habits and preferences.
-This allows users to gradually experience a sense of "understanding" when using the LLM.
-
-.. image:: docs/images/framework.png
- :align: center
-
-Framework
-^^^^^^^^^^^^^^^^^^^^
-
-💾 Memory Database: MemoryScope is equipped with a vector database (default is *ElasticSearch*) to store all memory fragments recorded in the system.
-
-🔧 Worker Library: MemoryScope atomizes the capabilities of long-term memory into individual workers, including over 20 workers for tasks such as query information filtering, observation extraction, and insight updating.
-
-🛠️ Operation Library: Based on the worker pipeline, it constructs the operations for memory services, realizing key capabilities such as memory retrieval and memory consolidation.
-
-- Memory Retrieval: Upon arrival of a user query, this operation returns the semantically related memory pieces
-and/or those from the corresponding time if the query involves reference to time.
-- Memory Consolidation: This operation takes in a batch of user queries and returns important user information
-extracted from the queries as consolidated *observations* to be stored in the memory database.
-- Reflection and Re-consolidation: At regular intervals, this operation performs reflection upon newly recorded *observations*
-to form and update *insights*. Then, memory re-consolidation is performed to ensure contradictions and repetitions
-among memory pieces are properly handled.
-
-.. toctree::
- :maxdepth: 2
- :caption: MemoryScope Tutorial
-
- About MemoryScope
- Installation
- Cli Client
- Simple Usages
- Advanced usage
- Contribution
-
-
-.. toctree::
- :maxdepth: 6
- :caption: MemoryScope API Reference
-
- API
diff --git a/memoryscope/docs/sphinx_doc/en/source/modules.rst b/memoryscope/docs/sphinx_doc/en/source/modules.rst
deleted file mode 100644
index dd0343a8..00000000
--- a/memoryscope/docs/sphinx_doc/en/source/modules.rst
+++ /dev/null
@@ -1,7 +0,0 @@
-memoryscope
-===========
-
-.. toctree::
- :maxdepth: 4
-
- memoryscope
diff --git a/memoryscope/docs/sphinx_doc/ja/source/index.rst b/memoryscope/docs/sphinx_doc/ja/source/index.rst
deleted file mode 100644
index 50cee814..00000000
--- a/memoryscope/docs/sphinx_doc/ja/source/index.rst
+++ /dev/null
@@ -1,55 +0,0 @@
-.. MemoryScope documentation master file, created by
- sphinx-quickstart on Fri Jan 5 17:53:54 2024.
- You can adapt this file completely to your liking, but it should at least
- contain the root `toctree` directive.
-
-:github_url: https://github.com/modelscope/memoryscope
-
-MemoryScope ドキュメント
-=========================
-
-MemoryScopeに関するドキュメントへようこそ
--------------------------------
-
-.. image:: ./docs/images/logo.png
- :align: center
-
-MemoryScopeは、LLMチャットボットに強力で柔軟な長期記憶能力を提供し、長期記憶能力を構築するためのフレームワークを提供します。
-MemoryScopeは、個人アシスタントや感情的な伴侶などの記憶シナリオに使用でき、長期記憶能力を通じてユーザーの基本情報やさまざまな習慣や好みを覚え続けることができます。
-これにより、ユーザーはLLMを使用する際に徐々に「理解されている」感覚を体験することができます。
-
-.. image:: docs/images/framework.png
- :align: center
-
-フレームワーク
-^^^^^^^^^^^^^^^^^^^^
-
-💾 メモリデータベース: MemoryScopeは、システム内に記録されたすべての記憶片を保存するためのベクトルデータベース（デフォルトは*ElasticSearch*）を備えています。
-
-🔧 ワーカーライブラリ: MemoryScopeは、長期記憶の能力を個々のワーカーに原子化し、クエリ情報のフィルタリング、観察の抽出、洞察の更新など、20以上のワーカーを含みます。
-
-🛠️ オペレーションライブラリ: ワーカーパイプラインに基づいて、メモリサービスのオペレーションを構築し、メモリの取得やメモリの統合などの主要な機能を実現します。
-
-- メモリの取得: ユーザークエリが到着すると、この操作は意味的に関連する記憶片を返します。
- クエリが時間に言及している場合は、対応する時間の記憶片も返します。
-- メモリの統合: この操作は、一連のユーザークエリを受け取り、クエリから抽出された重要なユーザー情報を統合された*観察*としてメモリデータベースに保存します。
-- 反映と再統合: 定期的に、この操作は新たに記録された*観察*を反映し、*洞察*を形成および更新します。
- その後、メモリの再統合を実行して、記憶片間の矛盾や重複が適切に処理されるようにします。
-
-.. toctree::
- :maxdepth: 2
- :caption: MemoryScope チュートリアル
-
- MemoryScopeについて
- インストール
- CLIクライアント
- 簡単な使用法
- 高度な使用法
- 貢献
-
-
-.. toctree::
- :maxdepth: 6
- :caption: MemoryScope APIリファレンス
-
- API
diff --git a/memoryscope/docs/sphinx_doc/requirements.txt b/memoryscope/docs/sphinx_doc/requirements.txt
deleted file mode 100644
index 96833f9d..00000000
--- a/memoryscope/docs/sphinx_doc/requirements.txt
+++ /dev/null
@@ -1,13 +0,0 @@
-loguru
-tiktoken
-pillow
-requests
-openai
-numpy
-sphinx
-sphinx-autobuild
-sphinx_rtd_theme
-sphinxcontrib-mermaid
-myst-parser
-autodoc_pydantic
-nbsphinx
diff --git a/memoryscope/docs/sphinx_doc/zh/source/_static/custom.css b/memoryscope/docs/sphinx_doc/zh/source/_static/custom.css
deleted file mode 100644
index 68f11cee..00000000
--- a/memoryscope/docs/sphinx_doc/zh/source/_static/custom.css
+++ /dev/null
@@ -1,4 +0,0 @@
-.language-selector a {
- color: white;
- width: 20px;
-}
\ No newline at end of file
diff --git a/memoryscope/docs/sphinx_doc/zh/source/_templates/language_selector.html b/memoryscope/docs/sphinx_doc/zh/source/_templates/language_selector.html
deleted file mode 100644
index 86fe0703..00000000
--- a/memoryscope/docs/sphinx_doc/zh/source/_templates/language_selector.html
+++ /dev/null
@@ -1,5 +0,0 @@
-
-
+
+
-
-
-