mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-08-28 05:25:04 +00:00
feat(tool): implement tool memory management and mock search tools- Added RetrieveToolMemoryOp for retrieving tool memory guidelines- Implemented LLM-based mock search tools with configurable complexity levels
- Enhanced tool call result parsing with improved scoring logic (0.0 or 1.0) - Updated tool memory schema to reflect binary success/failure scoring - Added deterministic behavior support via seed configuration in mock tools - Improved evaluation prompts to focus on result quality over success flags - Extended README with tool memory documentation and usage examples- Added utility functions for generating mock tool call results - Removed deprecated test file for UseMockSearchOp- Updated default configurations to include use_mock_search operation- Bumped version to0.1.10 and updated flowllm dependency requirement - Moved deprecation warnings to main init file - Simplified tool memory summary formatting by removing redundant statistics - Fixed tool call result processing to handle multiple tool names concurrently
This commit is contained in:
jinli.yl
parent
12f0b7d124
commit
49819a8e54
23 changed files with 2386 additions and 2773 deletions
171
README.md
171
README.md
|
|
@ -21,15 +21,16 @@ ReMe provides AI agents with a unified memory system—enabling the ability to e
|
|||
users, tasks, and agents.
|
||||
|
||||
```
|
||||
Personal Memory + Task Memory = Agent Memory
|
||||
Personal Memory + Task Memory + Tool Memory = Agent Memory
|
||||
```
|
||||
|
||||
Personal memory helps "**understand user preferences**", while task memory helps agents "**perform better**".
|
||||
Personal memory helps "**understand user preferences**", task memory helps agents "**perform better**", and tool memory enables "**smarter tool usage**".
|
||||
|
||||
---
|
||||
|
||||
## 📰 Latest Updates
|
||||
|
||||
- **[2025-10]** 🔧 Tool Memory support is now available! Enables data-driven tool selection and parameter optimization through historical performance tracking. Check out the [Tool Memory Guide](docs/tool_memory/tool_memory.md) and [benchmark results](docs/tool_memory/tool_bench.md).
|
||||
- **[2025-09-25]** 🎉 ReMe is exploring the directions of tool memory and Personal Memory Application/Agent.
|
||||
- **[2025-09]** 🎉 ReMe v0.1.9 has been officially released, adding support for asynchronous operations. It has also been
|
||||
integrated into the memory service of agentscope-runtime.
|
||||
|
|
@ -54,7 +55,7 @@ Personal memory helps "**understand user preferences**", while task memory helps
|
|||
<img src="docs/figure/reme_structure.jpg" alt="ReMe Logo" width="100%">
|
||||
</p>
|
||||
|
||||
ReMe integrates two complementary memory capabilities:
|
||||
ReMe integrates three complementary memory capabilities:
|
||||
|
||||
#### 🧠 **Task Memory/Experience**
|
||||
|
||||
|
|
@ -78,6 +79,17 @@ Contextualized memory for specific users
|
|||
|
||||
Learn more about how to use personal memory from [personal memory](docs/personal_memory/personal_memory.md)
|
||||
|
||||
#### 🔧 **Tool Memory**
|
||||
|
||||
Data-driven tool selection and usage optimization
|
||||
|
||||
- **Historical Performance Tracking**: Success rates, execution times, and token costs from real usage
|
||||
- **LLM-as-Judge Evaluation**: Qualitative insights on why tools succeed or fail
|
||||
- **Parameter Optimization**: Learn optimal parameter configurations from successful calls
|
||||
- **Dynamic Guidelines**: Transform static tool descriptions into living, learned manuals
|
||||
|
||||
Learn more about how to use tool memory from [tool memory](docs/tool_memory/tool_memory.md)
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Installation
|
||||
|
|
@ -316,6 +328,140 @@ fetch("http://localhost:8002/retrieve_personal_memory", {
|
|||
|
||||
</details>
|
||||
|
||||
#### Tool Memory Management
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
# Record tool execution results
|
||||
response = requests.post("http://localhost:8002/add_tool_call_result", json={
|
||||
"workspace_id": "tool_workspace",
|
||||
"tool_call_results": [
|
||||
{
|
||||
"create_time": "2025-10-21 10:30:00",
|
||||
"tool_name": "web_search",
|
||||
"input": {"query": "Python asyncio tutorial", "max_results": 10},
|
||||
"output": "Found 10 relevant results...",
|
||||
"token_cost": 150,
|
||||
"success": True,
|
||||
"time_cost": 2.3
|
||||
}
|
||||
]
|
||||
})
|
||||
|
||||
# Generate usage guidelines from history
|
||||
response = requests.post("http://localhost:8002/summary_tool_memory", json={
|
||||
"workspace_id": "tool_workspace",
|
||||
"tool_names": "web_search"
|
||||
})
|
||||
|
||||
# Retrieve tool guidelines before use
|
||||
response = requests.post("http://localhost:8002/retrieve_tool_memory", json={
|
||||
"workspace_id": "tool_workspace",
|
||||
"tool_names": "web_search"
|
||||
})
|
||||
```
|
||||
|
||||
<details>
|
||||
<summary>curl version</summary>
|
||||
|
||||
```bash
|
||||
# Record tool execution results
|
||||
curl -X POST http://localhost:8002/add_tool_call_result \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"workspace_id": "tool_workspace",
|
||||
"tool_call_results": [
|
||||
{
|
||||
"create_time": "2025-10-21 10:30:00",
|
||||
"tool_name": "web_search",
|
||||
"input": {"query": "Python asyncio tutorial", "max_results": 10},
|
||||
"output": "Found 10 relevant results...",
|
||||
"token_cost": 150,
|
||||
"success": true,
|
||||
"time_cost": 2.3
|
||||
}
|
||||
]
|
||||
}'
|
||||
|
||||
# Generate usage guidelines from history
|
||||
curl -X POST http://localhost:8002/summary_tool_memory \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"workspace_id": "tool_workspace",
|
||||
"tool_names": "web_search"
|
||||
}'
|
||||
|
||||
# Retrieve tool guidelines before use
|
||||
curl -X POST http://localhost:8002/retrieve_tool_memory \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"workspace_id": "tool_workspace",
|
||||
"tool_names": "web_search"
|
||||
}'
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>Node.js version</summary>
|
||||
|
||||
```javascript
|
||||
// Record tool execution results
|
||||
fetch("http://localhost:8002/add_tool_call_result", {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
body: JSON.stringify({
|
||||
workspace_id: "tool_workspace",
|
||||
tool_call_results: [
|
||||
{
|
||||
create_time: "2025-10-21 10:30:00",
|
||||
tool_name: "web_search",
|
||||
input: {query: "Python asyncio tutorial", max_results: 10},
|
||||
output: "Found 10 relevant results...",
|
||||
token_cost: 150,
|
||||
success: true,
|
||||
time_cost: 2.3
|
||||
}
|
||||
]
|
||||
})
|
||||
})
|
||||
.then(response => response.json())
|
||||
.then(data => console.log(data));
|
||||
|
||||
// Generate usage guidelines from history
|
||||
fetch("http://localhost:8002/summary_tool_memory", {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
body: JSON.stringify({
|
||||
workspace_id: "tool_workspace",
|
||||
tool_names: "web_search"
|
||||
})
|
||||
})
|
||||
.then(response => response.json())
|
||||
.then(data => console.log(data));
|
||||
|
||||
// Retrieve tool guidelines before use
|
||||
fetch("http://localhost:8002/retrieve_tool_memory", {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
body: JSON.stringify({
|
||||
workspace_id: "tool_workspace",
|
||||
tool_names: "web_search"
|
||||
})
|
||||
})
|
||||
.then(response => response.json())
|
||||
.then(data => console.log(data));
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## 📦 Ready-to-Use Libraries
|
||||
|
|
@ -387,12 +533,29 @@ We tested ReMe on BFCL-V3 multi-turn-base (randomly split 50train/150val) using
|
|||
| without ReMe | 0.2472 | 0.2733 | 0.2922 |
|
||||
| with ReMe | 0.3061 **(+5.89%)** | 0.3500 **(+7.67%)** | 0.3888 **(+9.66%)** |
|
||||
|
||||
### 🛠️ [Tool Memory Benchmark](docs/tool_memory/tool_bench.md)
|
||||
|
||||
We evaluated Tool Memory effectiveness using a controlled benchmark with three mock search tools using Qwen3-30B-Instruct:
|
||||
|
||||
| Scenario | Avg Score | Improvement |
|
||||
|-----------------------|-----------|--------------------|
|
||||
| Train (No Memory) | 0.650 | - |
|
||||
| Test (No Memory) | 0.672 | Baseline |
|
||||
| **Test (With Memory)** | **0.772** | **+14.88%** |
|
||||
|
||||
**Key Findings:**
|
||||
- Tool Memory enables data-driven tool selection based on historical performance
|
||||
- Success rates improved by ~15% with learned parameter configurations
|
||||
- Consistent improvement across all epochs (9.90% → 17.39% → 17.13%)
|
||||
|
||||
You can find more details in [tool_bench.md](docs/tool_memory/tool_bench.md) and the implementation at [run_reme_tool_bench.py](cookbook/tool_memory/run_reme_tool_bench.py).
|
||||
|
||||
## 📚 Resources
|
||||
|
||||
- **[Quick Start](./cookbook/simple_demo)**: Get started quickly with practical examples
|
||||
- **[Vector Storage Setup](docs/vector_store_api_guide.md)**: Configure local/vector databases and usage
|
||||
- **[MCP Guide](docs/mcp_quick_start.md)**: Create MCP services
|
||||
- **[personal memory](docs/personal_memory)** & **[task memory](docs/task_memory)** : Operators used in personal memory and task memory, You can modify the config to customize the pipelines.
|
||||
- **[Personal Memory](docs/personal_memory)**, **[Task Memory](docs/task_memory)** & **[Tool Memory](docs/tool_memory)**: Operators used in personal memory, task memory and tool memory. You can modify the config to customize the pipelines.
|
||||
- **[Example Collection](./cookbook)**: Real use cases and best practices
|
||||
|
||||
---
|
||||
|
|
|
|||
238
cookbook/simple_demo/use_tool_memory_demo.py
Normal file
238
cookbook/simple_demo/use_tool_memory_demo.py
Normal file
|
|
@ -0,0 +1,238 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""Tool Memory Demo - 展示工具记忆的完整生命周期"""
|
||||
|
||||
import json
|
||||
import time
|
||||
from typing import List, Dict, Any, Optional
|
||||
|
||||
import requests
|
||||
from dotenv import load_dotenv
|
||||
from reme_ai.utils.tool_memory_utils import create_mock_tool_call_results
|
||||
|
||||
load_dotenv()
|
||||
|
||||
BASE_URL = "http://0.0.0.0:8002/"
|
||||
WORKSPACE_ID = "test_tool_memory_workspace"
|
||||
|
||||
|
||||
def api_call(endpoint: str, data: dict) -> Optional[Dict[str, Any]]:
|
||||
"""统一的API调用处理"""
|
||||
response = requests.post(f"{BASE_URL}{endpoint}", json=data)
|
||||
if response.status_code != 200:
|
||||
print(f"Error: {response.status_code} - {response.text}")
|
||||
return None
|
||||
return response.json()
|
||||
|
||||
|
||||
def delete_workspace() -> None:
|
||||
"""删除工作空间
|
||||
|
||||
curl example:
|
||||
curl -X POST http://0.0.0.0:8002/vector_store \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"workspace_id": "test_tool_memory_workspace",
|
||||
"action": "delete"
|
||||
}'
|
||||
"""
|
||||
result = api_call("vector_store", {"workspace_id": WORKSPACE_ID, "action": "delete"})
|
||||
if result:
|
||||
print(f"✓ Workspace '{WORKSPACE_ID}' deleted")
|
||||
|
||||
|
||||
def add_tool_call_results(tool_call_results: List[Dict[str, Any]]) -> bool:
|
||||
"""添加工具调用结果到记忆库
|
||||
|
||||
curl example:
|
||||
curl -X POST http://0.0.0.0:8002/add_tool_call_result \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"workspace_id": "test_tool_memory_workspace",
|
||||
"tool_call_results": [
|
||||
{
|
||||
"tool_name": "web_search",
|
||||
"tool_input": "Python tutorials",
|
||||
"tool_output": "Found 100 results about Python",
|
||||
"execution_time": 0.5
|
||||
}
|
||||
]
|
||||
}'
|
||||
"""
|
||||
# 统计不同的工具
|
||||
tool_names = set(r.get("tool_name") for r in tool_call_results)
|
||||
print(f"\n[ADD] {len(tool_call_results)} results for {len(tool_names)} tools: {', '.join(sorted(tool_names))}")
|
||||
|
||||
result = api_call("add_tool_call_result", {
|
||||
"workspace_id": WORKSPACE_ID,
|
||||
"tool_call_results": tool_call_results
|
||||
})
|
||||
if result:
|
||||
memory_list = result.get("metadata", {}).get("memory_list", [])
|
||||
print(f"✓ Added successfully, created/updated {len(memory_list)} tool memories")
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def summarize_tool_memory(tool_names: str) -> Optional[Dict[str, Any]]:
|
||||
"""总结工具使用模式
|
||||
|
||||
curl example:
|
||||
curl -X POST http://0.0.0.0:8002/summary_tool_memory \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"workspace_id": "test_tool_memory_workspace",
|
||||
"tool_names": "web_search,database_query"
|
||||
}'
|
||||
"""
|
||||
print(f"\n[SUMMARIZE] {tool_names}")
|
||||
result = api_call("summary_tool_memory", {
|
||||
"workspace_id": WORKSPACE_ID,
|
||||
"tool_names": tool_names
|
||||
})
|
||||
|
||||
if result:
|
||||
memory_list = result.get("metadata", {}).get("memory_list", [])
|
||||
print(f"✓ Summarized {len(memory_list)} tool memories")
|
||||
for memory in memory_list:
|
||||
print(f"\n{'=' * 60}")
|
||||
print(f"Tool: {memory.get('when_to_use', 'N/A')}")
|
||||
print(f"{'=' * 60}")
|
||||
print(memory.get('content', 'No content'))
|
||||
return result
|
||||
|
||||
|
||||
def retrieve_tool_memory(tool_names: str, save_to_file: bool = False) -> str:
|
||||
"""检索工具记忆
|
||||
|
||||
curl example:
|
||||
curl -X POST http://0.0.0.0:8002/retrieve_tool_memory \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"workspace_id": "test_tool_memory_workspace",
|
||||
"tool_names": "web_search"
|
||||
}'
|
||||
"""
|
||||
print(f"\n[RETRIEVE] {tool_names}")
|
||||
result = api_call("retrieve_tool_memory", {
|
||||
"workspace_id": WORKSPACE_ID,
|
||||
"tool_names": tool_names
|
||||
})
|
||||
|
||||
if not result:
|
||||
return ""
|
||||
|
||||
memory_list = result.get("metadata", {}).get("memory_list", [])
|
||||
if not memory_list:
|
||||
print("No memories found")
|
||||
return ""
|
||||
|
||||
print(f"✓ Retrieved {len(memory_list)} memories")
|
||||
|
||||
formatted_memories = []
|
||||
for memory in memory_list:
|
||||
content = f"\nTool: {memory.get('when_to_use', 'N/A')}\n" \
|
||||
f"Calls: {len(memory.get('tool_call_results', []))}\n" \
|
||||
f"{'-' * 60}\n{memory.get('content', 'No content')}\n"
|
||||
formatted_memories.append(content)
|
||||
print(content)
|
||||
|
||||
if save_to_file:
|
||||
with open("tool_memory.json", "w") as f:
|
||||
json.dump(memory_list, f, indent=2, ensure_ascii=False)
|
||||
print("✓ Saved to tool_memory.json")
|
||||
|
||||
return "\n".join(formatted_memories)
|
||||
|
||||
|
||||
def dump_memory(path: str = "./") -> None:
|
||||
"""导出记忆到磁盘
|
||||
|
||||
curl example:
|
||||
curl -X POST http://0.0.0.0:8002/vector_store \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"workspace_id": "test_tool_memory_workspace",
|
||||
"action": "dump",
|
||||
"path": "./"
|
||||
}'
|
||||
"""
|
||||
result = api_call("vector_store", {
|
||||
"workspace_id": WORKSPACE_ID,
|
||||
"action": "dump",
|
||||
"path": path
|
||||
})
|
||||
if result:
|
||||
print(f"✓ Memory dumped to {path}")
|
||||
|
||||
|
||||
def load_memory(path: str = "./") -> None:
|
||||
"""从磁盘加载记忆
|
||||
|
||||
curl example:
|
||||
curl -X POST http://0.0.0.0:8002/vector_store \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"workspace_id": "test_tool_memory_workspace",
|
||||
"action": "load",
|
||||
"path": "./"
|
||||
}'
|
||||
"""
|
||||
result = api_call("vector_store", {
|
||||
"workspace_id": WORKSPACE_ID,
|
||||
"action": "load",
|
||||
"path": path
|
||||
})
|
||||
if result:
|
||||
print(f"✓ Memory loaded from {path}")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
# 1. 清理工作空间
|
||||
print("\n[1] Cleaning workspace...")
|
||||
delete_workspace()
|
||||
time.sleep(1)
|
||||
|
||||
# 2. 创建和添加模拟工具调用结果
|
||||
print("\n[2] Adding mock tool call results...")
|
||||
tools_to_test = [
|
||||
("web_search", 30),
|
||||
("database_query", 22),
|
||||
("file_processor", 18)
|
||||
]
|
||||
|
||||
# 收集所有工具的结果，然后一次性添加
|
||||
all_mock_results = []
|
||||
for tool_name, count in tools_to_test:
|
||||
mock_results = create_mock_tool_call_results(tool_name, count)
|
||||
all_mock_results.extend(mock_results)
|
||||
|
||||
if not add_tool_call_results(all_mock_results):
|
||||
print("✗ Failed to add results")
|
||||
else:
|
||||
time.sleep(1)
|
||||
|
||||
# 3. 总结工具记忆
|
||||
print("\n[3] Summarizing tool memories...")
|
||||
all_tool_names = ",".join([tool[0] for tool in tools_to_test])
|
||||
summarize_tool_memory(all_tool_names)
|
||||
time.sleep(1)
|
||||
|
||||
# 4. 检索工具记忆
|
||||
print("\n[4] Retrieving tool memories...")
|
||||
for tool_name, _ in tools_to_test:
|
||||
retrieve_tool_memory(tool_name, save_to_file=True)
|
||||
time.sleep(0.5)
|
||||
|
||||
# 5. 测试记忆持久化
|
||||
print("\n[5] Testing memory persistence...")
|
||||
dump_memory()
|
||||
load_memory()
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("DEMO COMPLETE ✓")
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
633
cookbook/tool_memory/run_reme_tool_bench.py
Normal file
633
cookbook/tool_memory/run_reme_tool_bench.py
Normal file
|
|
@ -0,0 +1,633 @@
|
|||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
Tool Memory Benchmark Script
|
||||
评估工具记忆在不同场景下的效果，包括有记忆和无记忆的对比
|
||||
|
||||
Dependencies:
|
||||
pip install requests python-dotenv loguru tabulate
|
||||
"""
|
||||
|
||||
import json
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from pathlib import Path
|
||||
from typing import List, Dict, Any, Optional
|
||||
|
||||
import requests
|
||||
from dotenv import load_dotenv
|
||||
from loguru import logger
|
||||
from tabulate import tabulate
|
||||
|
||||
from reme_ai.schema.memory import ToolCallResult, ToolMemory
|
||||
|
||||
load_dotenv()
|
||||
|
||||
BASE_URL = "http://0.0.0.0:8002/"
|
||||
TRAIN_WORKSPACE = "train_tool_workspace"
|
||||
TEST_WORKSPACE = "test_tool_workspace"
|
||||
|
||||
|
||||
class BenchmarkStats:
|
||||
"""统计数据收集器"""
|
||||
|
||||
def __init__(self, name: str):
|
||||
self.name = name
|
||||
self.total_count = 0
|
||||
self.scores = []
|
||||
|
||||
def add_result(self, result: Dict[str, Any]):
|
||||
"""添加一个工具调用结果
|
||||
|
||||
Note:
|
||||
- score: Quality/relevance of the result (0.0 or 1.0)
|
||||
"""
|
||||
self.total_count += 1
|
||||
|
||||
# Collect score
|
||||
score = result.get("score", 0)
|
||||
self.scores.append(score)
|
||||
|
||||
def get_summary(self) -> Dict[str, Any]:
|
||||
"""获取统计摘要"""
|
||||
if self.total_count == 0:
|
||||
return {
|
||||
"name": self.name,
|
||||
"total_calls": 0,
|
||||
"avg_score": 0.0
|
||||
}
|
||||
|
||||
return {
|
||||
"name": self.name,
|
||||
"total_calls": self.total_count,
|
||||
"avg_score": round(sum(self.scores) / len(self.scores), 3)
|
||||
}
|
||||
|
||||
|
||||
def api_call(endpoint: str, data: dict) -> Optional[Dict[str, Any]]:
|
||||
"""统一的API调用处理"""
|
||||
try:
|
||||
response = requests.post(f"{BASE_URL}{endpoint}", json=data, timeout=120)
|
||||
if response.status_code != 200:
|
||||
logger.error(f"API Error: {response.status_code} - {response.text}")
|
||||
return None
|
||||
return response.json()
|
||||
except Exception as e:
|
||||
logger.error(f"API call failed: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def delete_workspace(workspace_id: str) -> bool:
|
||||
"""删除工作空间"""
|
||||
logger.info(f"Deleting workspace: {workspace_id}")
|
||||
result = api_call("vector_store", {"workspace_id": workspace_id, "action": "delete"})
|
||||
return result is not None
|
||||
|
||||
|
||||
def load_queries(query_file: str = "query.json") -> Dict[str, Any]:
|
||||
"""加载查询数据"""
|
||||
query_path = Path(__file__).parent / query_file
|
||||
with open(query_path, 'r', encoding='utf-8') as f:
|
||||
return json.load(f)
|
||||
|
||||
|
||||
def run_use_mock_search(workspace_id: str, queries: List[str], prompt_template: str = "") -> List[ToolCallResult]:
|
||||
"""运行use_mock_search并收集结果（支持并发）
|
||||
|
||||
Args:
|
||||
workspace_id: 工作空间ID
|
||||
queries: 查询列表
|
||||
prompt_template: 提示模板
|
||||
|
||||
Returns:
|
||||
工具调用结果列表
|
||||
"""
|
||||
logger.info(f"Running use_mock_search on {workspace_id} with {len(queries)} queries (max concurrency: 4)")
|
||||
results: List[ToolCallResult] = []
|
||||
|
||||
def process_single_query(idx: int, query: str) -> Optional[ToolCallResult]:
|
||||
"""处理单个查询"""
|
||||
logger.info(f"[{idx + 1}/{len(queries)}] Processing: {query}")
|
||||
|
||||
# 提交之前sleep 1秒
|
||||
time.sleep(1)
|
||||
|
||||
result = api_call("use_mock_search", {
|
||||
"workspace_id": workspace_id,
|
||||
"query": prompt_template.format(query=query),
|
||||
})
|
||||
|
||||
if result:
|
||||
tool_call_result = result.get("answer")
|
||||
return ToolCallResult(**json.loads(tool_call_result))
|
||||
else:
|
||||
logger.warning(f"No result for query: {query}")
|
||||
return None
|
||||
|
||||
# 使用线程池并发处理，最大并发数为4
|
||||
with ThreadPoolExecutor(max_workers=4) as executor:
|
||||
# 提交所有任务
|
||||
future_to_query = {
|
||||
executor.submit(process_single_query, idx, query): (idx, query)
|
||||
for idx, query in enumerate(queries)
|
||||
}
|
||||
|
||||
# 收集结果（按完成顺序）
|
||||
completed_results = []
|
||||
for future in as_completed(future_to_query):
|
||||
idx, query = future_to_query[future]
|
||||
try:
|
||||
result = future.result()
|
||||
if result:
|
||||
completed_results.append((idx, result))
|
||||
except Exception as e:
|
||||
logger.error(f"Error processing query [{idx + 1}]: {query}, error: {e}")
|
||||
|
||||
# 按原始顺序排序结果
|
||||
completed_results.sort(key=lambda x: x[0])
|
||||
results = [result for _, result in completed_results]
|
||||
|
||||
logger.info(f"Collected {len(results)} results out of {len(queries)} queries")
|
||||
return results
|
||||
|
||||
|
||||
def add_tool_call_results(workspace_id: str, results: List[ToolCallResult]) -> List[ToolCallResult]:
|
||||
"""批量添加工具调用结果到记忆库，并返回带评分的结果
|
||||
|
||||
Args:
|
||||
workspace_id: 工作空间ID
|
||||
results: 工具调用结果列表
|
||||
|
||||
Returns:
|
||||
从API返回的memory_list中提取的带评分的ToolCallResult列表
|
||||
"""
|
||||
if not results:
|
||||
logger.warning("No results to add")
|
||||
return []
|
||||
|
||||
# 转换为字典用于API调用
|
||||
tool_call_results = [result.model_dump() for result in results]
|
||||
|
||||
logger.info(f"Adding tool call results to {workspace_id}: {len(tool_call_results)} results")
|
||||
|
||||
# 统一调用API，让后端自动按tool_name分组处理
|
||||
api_result = api_call("add_tool_call_result", {
|
||||
"workspace_id": workspace_id,
|
||||
"tool_call_results": tool_call_results
|
||||
})
|
||||
|
||||
if not api_result:
|
||||
logger.error("Failed to add results")
|
||||
return []
|
||||
|
||||
# 收集所有带评分的结果
|
||||
all_scored_results: List[ToolCallResult] = []
|
||||
|
||||
# 解析返回的memory_list（可能包含多个工具的记忆）
|
||||
memory_list = api_result.get("metadata", {}).get("memory_list", [])
|
||||
logger.info(f"Received {len(memory_list)} tool memories from API")
|
||||
|
||||
for memory_dict in memory_list:
|
||||
tool_memory = ToolMemory(**memory_dict)
|
||||
tool_name = tool_memory.when_to_use
|
||||
scored_results = tool_memory.tool_call_results
|
||||
all_scored_results.extend(scored_results)
|
||||
|
||||
logger.info(f"Extracted {len(scored_results)} scored results from {tool_name}")
|
||||
|
||||
# 打印一些评分示例
|
||||
for idx, result in enumerate(scored_results[:3]):
|
||||
logger.info(f" Result #{idx + 1}: score={result.score}, success={result.success}")
|
||||
|
||||
logger.info(f"Total scored results collected: {len(all_scored_results)}")
|
||||
return all_scored_results
|
||||
|
||||
|
||||
def summarize_tool_memory(workspace_id: str, tool_names: str) -> bool:
|
||||
"""总结工具记忆"""
|
||||
logger.info(f"Summarizing tool memory for {workspace_id}: {tool_names}")
|
||||
result = api_call("summary_tool_memory", {
|
||||
"workspace_id": workspace_id,
|
||||
"tool_names": tool_names
|
||||
})
|
||||
|
||||
if result:
|
||||
memory_list = result.get("metadata", {}).get("memory_list", [])
|
||||
logger.info(f"Summarized {len(memory_list)} tool memories")
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def retrieve_tool_memory(workspace_id: str, tool_names: str) -> str:
|
||||
"""检索工具记忆并返回格式化的内容
|
||||
|
||||
Args:
|
||||
workspace_id: 工作空间ID
|
||||
tool_names: 逗号分隔的工具名称
|
||||
|
||||
Returns:
|
||||
格式化的工具记忆内容，每个工具名称作为一级markdown标题
|
||||
"""
|
||||
logger.info(f"Retrieving tool memory for {workspace_id}: {tool_names}")
|
||||
result = api_call("retrieve_tool_memory", {
|
||||
"workspace_id": workspace_id,
|
||||
"tool_names": tool_names
|
||||
})
|
||||
|
||||
if not result:
|
||||
logger.error("Failed to retrieve tool memory")
|
||||
return ""
|
||||
|
||||
memory_list = result.get("metadata", {}).get("memory_list", [])
|
||||
logger.info(f"Retrieved {len(memory_list)} tool memories")
|
||||
|
||||
# 提取每个工具记忆的content字段，并格式化为markdown
|
||||
formatted_contents = []
|
||||
for memory_dict in memory_list:
|
||||
tool_memory = ToolMemory(**memory_dict)
|
||||
if tool_memory.content:
|
||||
# 使用工具名称作为一级markdown标题
|
||||
tool_name = tool_memory.when_to_use or "Unknown Tool"
|
||||
formatted_section = f"# {tool_name}\n\n{tool_memory.content}"
|
||||
formatted_contents.append(formatted_section)
|
||||
logger.info(f"Retrieved content for tool: {tool_name}, "
|
||||
f"content_length={len(tool_memory.content)}")
|
||||
|
||||
# 用两个换行符分隔不同工具的记忆
|
||||
joined_content = "\n\n".join(formatted_contents)
|
||||
logger.info(f"Total content length: {len(joined_content)}")
|
||||
|
||||
return joined_content
|
||||
|
||||
|
||||
def collect_statistics(results: List[ToolCallResult], stats: BenchmarkStats) -> None:
|
||||
"""从结果列表中收集统计数据"""
|
||||
for result in results:
|
||||
# 转换为字典用于统计
|
||||
result_dict = result.model_dump() if hasattr(result, 'model_dump') else result
|
||||
stats.add_result(result_dict)
|
||||
|
||||
|
||||
def print_comparison_table(stats_list: List[BenchmarkStats]) -> None:
|
||||
"""打印对比表格"""
|
||||
headers = ["Scenario", "Total Calls", "Avg Score"]
|
||||
rows = []
|
||||
|
||||
for stats in stats_list:
|
||||
summary = stats.get_summary()
|
||||
rows.append([
|
||||
summary["name"],
|
||||
summary["total_calls"],
|
||||
summary["avg_score"]
|
||||
])
|
||||
|
||||
print("\n" + "=" * 100)
|
||||
print("BENCHMARK RESULTS COMPARISON")
|
||||
print("=" * 100)
|
||||
print("Note: Avg Score = average quality score")
|
||||
print(tabulate(rows, headers=headers, tablefmt="grid"))
|
||||
print("=" * 100)
|
||||
|
||||
|
||||
def calculate_improvements(baseline_stats: BenchmarkStats, improved_stats: BenchmarkStats) -> Dict[str, float]:
|
||||
"""计算改进百分比"""
|
||||
baseline = baseline_stats.get_summary()
|
||||
improved = improved_stats.get_summary()
|
||||
|
||||
improvements = {}
|
||||
|
||||
# 平均分数改进（相对提升百分比）
|
||||
if baseline["avg_score"] > 0:
|
||||
improvements["avg_score"] = ((improved["avg_score"] - baseline["avg_score"])
|
||||
/ baseline["avg_score"] * 100)
|
||||
else:
|
||||
improvements["avg_score"] = 0.0
|
||||
|
||||
return improvements
|
||||
|
||||
|
||||
def print_improvements(improvements: Dict[str, float]) -> None:
|
||||
"""打印改进情况"""
|
||||
print("\n" + "=" * 100)
|
||||
print("IMPROVEMENTS WITH TOOL MEMORY (Baseline: Test without memory)")
|
||||
print("=" * 100)
|
||||
|
||||
metric_labels = {
|
||||
"avg_score": "Average Score"
|
||||
}
|
||||
|
||||
for metric, improvement in improvements.items():
|
||||
label = metric_labels.get(metric, metric)
|
||||
direction = "↑" if improvement > 0 else "↓"
|
||||
print(f"{label:25s}: {improvement:+7.2f}% {direction}")
|
||||
|
||||
print("=" * 100)
|
||||
|
||||
|
||||
def save_results(results: Dict[str, Any], filename: str = "benchmark_results.json") -> None:
|
||||
"""保存结果到文件"""
|
||||
output_path = Path(__file__).parent / filename
|
||||
with open(output_path, 'w', encoding='utf-8') as f:
|
||||
json.dump(results, f, indent=2, ensure_ascii=False)
|
||||
logger.info(f"Results saved to {output_path}")
|
||||
|
||||
|
||||
def run_single_epoch(epoch_num: int, train_queries: List[str], test_queries: List[str]) -> Dict[str, Any]:
|
||||
"""运行单个epoch的benchmark
|
||||
|
||||
Args:
|
||||
epoch_num: epoch编号（从1开始）
|
||||
train_queries: 训练查询列表
|
||||
test_queries: 测试查询列表
|
||||
|
||||
Returns:
|
||||
包含该epoch统计结果的字典
|
||||
"""
|
||||
logger.info("\n" + "=" * 100)
|
||||
logger.info(f"EPOCH {epoch_num} - START")
|
||||
logger.info("=" * 100)
|
||||
|
||||
# 初始化统计收集器
|
||||
train_no_memory_stats = BenchmarkStats(f"Epoch{epoch_num} - Train (No Memory)")
|
||||
test_no_memory_stats = BenchmarkStats(f"Epoch{epoch_num} - Test (No Memory)")
|
||||
test_with_memory_stats = BenchmarkStats(f"Epoch{epoch_num} - Test (With Memory)")
|
||||
|
||||
all_results = {}
|
||||
|
||||
# ==================== 步骤1: 无记忆在train上的效果 ====================
|
||||
print("\n" + "=" * 100)
|
||||
print(f"[EPOCH {epoch_num}] [STEP 1/5] Running on TRAIN without memory...")
|
||||
print("=" * 100)
|
||||
logger.info("Deleting workspace and starting fresh...")
|
||||
delete_workspace(TRAIN_WORKSPACE)
|
||||
time.sleep(2)
|
||||
prompt_template = "必须选择一个工具来回答问题\n 问题\n{query}"
|
||||
train_results_no_memory = run_use_mock_search(TRAIN_WORKSPACE, train_queries, prompt_template)
|
||||
|
||||
# 添加结果到记忆库并获取带评分的结果
|
||||
train_scored_results = add_tool_call_results(TRAIN_WORKSPACE, train_results_no_memory)
|
||||
time.sleep(2)
|
||||
|
||||
# 使用带评分的结果进行统计（如果有的话）
|
||||
if train_scored_results:
|
||||
logger.info(f"Using {len(train_scored_results)} scored results for statistics")
|
||||
all_results["train_no_memory"] = train_scored_results
|
||||
collect_statistics(train_scored_results, train_no_memory_stats)
|
||||
else:
|
||||
logger.warning("No scored results returned, using original results")
|
||||
all_results["train_no_memory"] = train_results_no_memory
|
||||
collect_statistics(train_results_no_memory, train_no_memory_stats)
|
||||
|
||||
print(f"✓ Train (no memory) completed: {len(train_results_no_memory)}/{len(train_queries)} results collected")
|
||||
summary = train_no_memory_stats.get_summary()
|
||||
print(f" Avg Score: {summary['avg_score']:.3f}")
|
||||
|
||||
# ==================== 步骤2: 无记忆在test上的效果 ====================
|
||||
print("\n" + "=" * 100)
|
||||
print(f"[EPOCH {epoch_num}] [STEP 2/5] Running on TEST without memory...")
|
||||
print("=" * 100)
|
||||
logger.info("Deleting workspace and starting fresh...")
|
||||
delete_workspace(TEST_WORKSPACE)
|
||||
time.sleep(2)
|
||||
|
||||
prompt_template = "必须选择一个工具来回答问题\n 问题\n{query}"
|
||||
test_results_no_memory = run_use_mock_search(TEST_WORKSPACE, test_queries, prompt_template)
|
||||
|
||||
# 添加结果到记忆库并获取带评分的结果
|
||||
# 注意：这些结果会作为TEST_WORKSPACE的初始记忆，在步骤4会被复用
|
||||
test_scored_results_no_memory = add_tool_call_results(TEST_WORKSPACE, test_results_no_memory)
|
||||
time.sleep(2)
|
||||
|
||||
# 使用带评分的结果进行统计（如果有的话）
|
||||
if test_scored_results_no_memory:
|
||||
logger.info(f"Using {len(test_scored_results_no_memory)} scored results for statistics")
|
||||
all_results["test_no_memory"] = test_scored_results_no_memory
|
||||
collect_statistics(test_scored_results_no_memory, test_no_memory_stats)
|
||||
else:
|
||||
logger.warning("No scored results returned, using original results")
|
||||
all_results["test_no_memory"] = test_results_no_memory
|
||||
collect_statistics(test_results_no_memory, test_no_memory_stats)
|
||||
|
||||
print(f"✓ Test (no memory) completed: {len(test_results_no_memory)}/{len(test_queries)} results collected")
|
||||
summary = test_no_memory_stats.get_summary()
|
||||
print(f" Avg Score: {summary['avg_score']:.3f}")
|
||||
|
||||
# ==================== 步骤3: 总结train的工具记忆 ====================
|
||||
print("\n" + "=" * 100)
|
||||
print(f"[EPOCH {epoch_num}] [STEP 3/5] Summarizing tool memory from TRAIN...")
|
||||
print("=" * 100)
|
||||
|
||||
# 获取所有工具名称（使用带评分的结果）
|
||||
tool_names_set = set()
|
||||
results_to_use = train_scored_results if train_scored_results else train_results_no_memory
|
||||
for result in results_to_use:
|
||||
tool_name = result.tool_name if hasattr(result, 'tool_name') else None
|
||||
if tool_name:
|
||||
tool_names_set.add(tool_name)
|
||||
|
||||
tool_names_str = ",".join(sorted(tool_names_set))
|
||||
print(f"Tools to summarize: {tool_names_str}")
|
||||
|
||||
success = summarize_tool_memory(TRAIN_WORKSPACE, tool_names_str)
|
||||
if not success:
|
||||
logger.error("Failed to summarize tool memory")
|
||||
return {}
|
||||
|
||||
time.sleep(3)
|
||||
|
||||
print("✓ Tool memory summarized successfully")
|
||||
|
||||
# 检索工具记忆内容
|
||||
memories = retrieve_tool_memory(TRAIN_WORKSPACE, tool_names_str)
|
||||
if not memories:
|
||||
logger.error("Failed to retrieve tool memory content")
|
||||
return {}
|
||||
|
||||
logger.info(f"Retrieved tool memory content, total length: {len(memories)}")
|
||||
print("\n" + "-" * 100)
|
||||
print("Retrieved Tool Memory Content:")
|
||||
print("-" * 100)
|
||||
print(memories)
|
||||
print("-" * 100)
|
||||
|
||||
# ==================== 步骤4: 有记忆在test上的效果 ====================
|
||||
print("\n" + "=" * 100)
|
||||
print(f"[EPOCH {epoch_num}] [STEP 4/5] Running on TEST with memory (after clearing existing memory)...")
|
||||
print("=" * 100)
|
||||
|
||||
# 先清理TEST_WORKSPACE中已有的记忆记录（Step 2的60条结果）
|
||||
print("Deleting existing memory records from TEST workspace...")
|
||||
delete_workspace(TEST_WORKSPACE)
|
||||
time.sleep(2) # 等待删除完成
|
||||
print("✓ TEST workspace memory cleared")
|
||||
|
||||
# 通过prompt注入train阶段总结的记忆，测量记忆增强的效果
|
||||
# 注意：此时workspace是空的，只通过prompt提供记忆信息
|
||||
|
||||
prompt_template = f"工具信息\n{memories}\n必须选择一个工具来回答问题\n 问题\n" + "{query}"
|
||||
test_results_with_memory = run_use_mock_search(TEST_WORKSPACE, test_queries, prompt_template)
|
||||
|
||||
# 添加这些结果到记忆库并获取带评分的结果
|
||||
# 此时workspace已清空，返回的就是本次新增的60条结果
|
||||
test_all_results_with_memory = add_tool_call_results(TEST_WORKSPACE, test_results_with_memory)
|
||||
time.sleep(2)
|
||||
|
||||
# workspace已清空，所有返回的结果都是本次新增的
|
||||
if test_all_results_with_memory:
|
||||
test_scored_results_with_memory = test_all_results_with_memory
|
||||
logger.info(f"Using {len(test_scored_results_with_memory)} scored results for statistics")
|
||||
all_results["test_with_memory"] = test_scored_results_with_memory
|
||||
collect_statistics(test_scored_results_with_memory, test_with_memory_stats)
|
||||
else:
|
||||
logger.warning("No scored results returned, using original results")
|
||||
all_results["test_with_memory"] = test_results_with_memory
|
||||
collect_statistics(test_results_with_memory, test_with_memory_stats)
|
||||
|
||||
print(f"✓ Test (with memory) completed: {len(test_results_with_memory)}/{len(test_queries)} results collected")
|
||||
summary = test_with_memory_stats.get_summary()
|
||||
print(f" Avg Score: {summary['avg_score']:.3f}")
|
||||
|
||||
# ==================== 步骤5: 打印对比结果 ====================
|
||||
print("\n" + "=" * 100)
|
||||
print(f"[EPOCH {epoch_num}] [STEP 5/5] Generating comparison report and analysis...")
|
||||
print("=" * 100)
|
||||
|
||||
# 打印统计表格
|
||||
print_comparison_table([train_no_memory_stats, test_no_memory_stats, test_with_memory_stats])
|
||||
|
||||
# 计算并打印改进情况
|
||||
improvements = calculate_improvements(test_no_memory_stats, test_with_memory_stats)
|
||||
print_improvements(improvements)
|
||||
|
||||
logger.info(f"EPOCH {epoch_num} - COMPLETE")
|
||||
|
||||
return {
|
||||
"epoch": epoch_num,
|
||||
"statistics": {
|
||||
"train_no_memory": train_no_memory_stats.get_summary(),
|
||||
"test_no_memory": test_no_memory_stats.get_summary(),
|
||||
"test_with_memory": test_with_memory_stats.get_summary()
|
||||
},
|
||||
"improvements": improvements
|
||||
}
|
||||
|
||||
|
||||
def main(test_mode: bool = False, run_epoch: int = 3):
|
||||
"""主函数：运行完整的benchmark流程
|
||||
|
||||
Args:
|
||||
test_mode: 如果为True，只使用每个难度级别的前3个查询进行快速测试
|
||||
run_epoch: 运行的epoch数量，默认为3
|
||||
"""
|
||||
logger.info("=" * 100)
|
||||
logger.info("TOOL MEMORY BENCHMARK - START")
|
||||
if test_mode:
|
||||
logger.info("Running in TEST MODE (limited queries)")
|
||||
logger.info(f"Total Epochs: {run_epoch}")
|
||||
logger.info("=" * 100)
|
||||
|
||||
# 加载查询数据
|
||||
queries_data = load_queries()
|
||||
train_queries = []
|
||||
test_queries = []
|
||||
|
||||
# 合并所有难度级别的查询
|
||||
for difficulty in ["simple", "moderate", "complex"]:
|
||||
train_data = queries_data["train"].get(difficulty, [])
|
||||
test_data = queries_data["test"].get(difficulty, [])
|
||||
|
||||
if test_mode:
|
||||
train_queries.extend(train_data[:5])
|
||||
test_queries.extend(test_data[:5])
|
||||
else:
|
||||
train_queries.extend(train_data)
|
||||
test_queries.extend(test_data)
|
||||
|
||||
logger.info(f"Loaded {len(train_queries)} train queries and {len(test_queries)} test queries")
|
||||
|
||||
# 运行多个epoch并收集结果
|
||||
all_epoch_results = []
|
||||
|
||||
for epoch in range(1, run_epoch + 1):
|
||||
epoch_result = run_single_epoch(epoch, train_queries, test_queries)
|
||||
if epoch_result:
|
||||
all_epoch_results.append(epoch_result)
|
||||
else:
|
||||
logger.error(f"Epoch {epoch} failed, skipping...")
|
||||
|
||||
# ==================== 计算多轮平均效果 ====================
|
||||
if not all_epoch_results:
|
||||
logger.error("No successful epochs, cannot calculate averages")
|
||||
return
|
||||
|
||||
print("\n" + "=" * 100)
|
||||
print("MULTI-EPOCH AVERAGE RESULTS")
|
||||
print("=" * 100)
|
||||
|
||||
# 计算每个场景的平均分数
|
||||
avg_train_no_memory = sum(e["statistics"]["train_no_memory"]["avg_score"] for e in all_epoch_results) / len(
|
||||
all_epoch_results)
|
||||
avg_test_no_memory = sum(e["statistics"]["test_no_memory"]["avg_score"] for e in all_epoch_results) / len(
|
||||
all_epoch_results)
|
||||
avg_test_with_memory = sum(e["statistics"]["test_with_memory"]["avg_score"] for e in all_epoch_results) / len(
|
||||
all_epoch_results)
|
||||
|
||||
# 计算平均改进
|
||||
avg_improvement = sum(e["improvements"]["avg_score"] for e in all_epoch_results) / len(all_epoch_results)
|
||||
|
||||
# 打印汇总表格
|
||||
headers = ["Scenario", "Avg Score (across epochs)"]
|
||||
rows = [
|
||||
["Train (No Memory)", f"{avg_train_no_memory:.3f}"],
|
||||
["Test (No Memory)", f"{avg_test_no_memory:.3f}"],
|
||||
["Test (With Memory)", f"{avg_test_with_memory:.3f}"]
|
||||
]
|
||||
print(tabulate(rows, headers=headers, tablefmt="grid"))
|
||||
|
||||
print("\n" + "-" * 100)
|
||||
print(f"Average Improvement (Test with memory vs without): {avg_improvement:+.2f}%")
|
||||
print("-" * 100)
|
||||
|
||||
# 打印每个epoch的详细结果
|
||||
print("\n" + "=" * 100)
|
||||
print("PER-EPOCH BREAKDOWN")
|
||||
print("=" * 100)
|
||||
|
||||
headers = ["Epoch", "Train (No Mem)", "Test (No Mem)", "Test (With Mem)", "Improvement %"]
|
||||
rows = []
|
||||
for e in all_epoch_results:
|
||||
rows.append([
|
||||
f"Epoch {e['epoch']}",
|
||||
f"{e['statistics']['train_no_memory']['avg_score']:.3f}",
|
||||
f"{e['statistics']['test_no_memory']['avg_score']:.3f}",
|
||||
f"{e['statistics']['test_with_memory']['avg_score']:.3f}",
|
||||
f"{e['improvements']['avg_score']:+.2f}%"
|
||||
])
|
||||
|
||||
print(tabulate(rows, headers=headers, tablefmt="grid"))
|
||||
|
||||
# 保存最终结果
|
||||
benchmark_summary = {
|
||||
"timestamp": time.strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"total_epochs": run_epoch,
|
||||
"successful_epochs": len(all_epoch_results),
|
||||
"average_results": {
|
||||
"train_no_memory": avg_train_no_memory,
|
||||
"test_no_memory": avg_test_no_memory,
|
||||
"test_with_memory": avg_test_with_memory,
|
||||
"improvement": avg_improvement
|
||||
},
|
||||
"per_epoch_results": all_epoch_results
|
||||
}
|
||||
|
||||
save_results(benchmark_summary, "tool_memory_benchmark_results.json")
|
||||
|
||||
logger.info("\n" + "=" * 100)
|
||||
logger.info("TOOL MEMORY BENCHMARK - COMPLETE")
|
||||
logger.info(f"Successfully completed {len(all_epoch_results)}/{run_epoch} epochs")
|
||||
logger.info("=" * 100)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main(test_mode=False, run_epoch=3)
|
||||
File diff suppressed because it is too large
Load diff
File diff suppressed because it is too large
Load diff
|
|
@ -12,537 +12,8 @@ Retrieves tool memories from the vector database based on tool names, providing
|
|||
- Searches the vector store for exact tool name matches
|
||||
- Validates that retrieved memories are of type "tool"
|
||||
- Returns complete tool memories including usage guidelines and call history
|
||||
- Provides detailed logging for debugging and monitoring
|
||||
|
||||
### Processing Flow
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
A[Receive Tool Names] --> B[Validate Input]
|
||||
B --> C[Split by Comma]
|
||||
C --> D[Trim Whitespace]
|
||||
D --> E[For Each Tool Name]
|
||||
E --> F[Search Vector Store]
|
||||
F --> G{Results Found?}
|
||||
G -->|Yes| H[Get Top Result]
|
||||
G -->|No| I[Log Warning: Not Found]
|
||||
H --> J{Type = tool?}
|
||||
J -->|Yes| K{Name Matches?}
|
||||
J -->|No| L[Log Warning: Wrong Type]
|
||||
K -->|Yes| M[Add to Results]
|
||||
K -->|No| N[Log Warning: Name Mismatch]
|
||||
I --> O[Continue Next Tool]
|
||||
L --> O
|
||||
N --> O
|
||||
M --> O
|
||||
O --> P{More Tools?}
|
||||
P -->|Yes| E
|
||||
P -->|No| Q{Any Matches?}
|
||||
Q -->|Yes| R[Return Memory List]
|
||||
Q -->|No| S[Return Empty]
|
||||
```
|
||||
|
||||
1. **Input Validation**:
|
||||
- Check if `tool_names` parameter is provided
|
||||
- Return error if empty
|
||||
- Log workspace_id and tool count
|
||||
|
||||
2. **Tool Name Processing**:
|
||||
- Split input by comma delimiter
|
||||
- Strip whitespace from each tool name
|
||||
- Filter out empty strings
|
||||
- Log the list of tools to retrieve
|
||||
|
||||
3. **Vector Store Search**:
|
||||
- For each tool name, search with `top_k=1`
|
||||
- Use tool name as the query (exact match preferred)
|
||||
- Retrieve the top matching result
|
||||
|
||||
4. **Result Validation**:
|
||||
- Verify result is of type `ToolMemory`
|
||||
- Check that `when_to_use` field exactly matches tool name
|
||||
- Log match details (memory_id, total_calls)
|
||||
- Warn if no match or mismatch found
|
||||
|
||||
5. **Response Preparation**:
|
||||
- Collect all matched tool memories
|
||||
- Set success status based on matches found
|
||||
- Return memory list in metadata
|
||||
|
||||
### Parameters
|
||||
|
||||
This operation has no configurable parameters. It uses the default vector store configuration.
|
||||
|
||||
### Input Schema
|
||||
|
||||
```yaml
|
||||
input_schema:
|
||||
tool_names:
|
||||
type: string
|
||||
description: "Comma-separated tool names (e.g., 'tool_name1,tool_name2')"
|
||||
required: true
|
||||
```
|
||||
|
||||
### Output Format
|
||||
|
||||
The operation sets the following in `context.response`:
|
||||
|
||||
```python
|
||||
{
|
||||
"success": True,
|
||||
"answer": "Successfully retrieved 2 tool memories",
|
||||
"metadata": {
|
||||
"memory_list": [
|
||||
{
|
||||
"workspace_id": "demo_workspace",
|
||||
"memory_id": "abc123def456",
|
||||
"memory_type": "tool",
|
||||
"when_to_use": "web_search",
|
||||
"content": "Core Function: The web_search tool retrieves...",
|
||||
"score": 0.85,
|
||||
"time_created": "2025-10-15 10:00:00",
|
||||
"time_modified": "2025-10-15 14:30:00",
|
||||
"author": "qwen3-30b-a3b-instruct-2507",
|
||||
"tool_call_results": [
|
||||
{
|
||||
"create_time": "2025-10-15 14:30:00",
|
||||
"tool_name": "web_search",
|
||||
"input": {...},
|
||||
"output": "...",
|
||||
"summary": "...",
|
||||
"evaluation": "...",
|
||||
"score": 1.0,
|
||||
"success": True,
|
||||
"time_cost": 2.3,
|
||||
"token_cost": 150
|
||||
}
|
||||
],
|
||||
"metadata": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Usage Example
|
||||
|
||||
#### Basic Retrieval
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
BASE_URL = "http://0.0.0.0:8002/"
|
||||
WORKSPACE_ID = "demo_workspace"
|
||||
|
||||
# Retrieve memory for a single tool
|
||||
response = requests.post(
|
||||
url=f"{BASE_URL}retrieve_tool_memory",
|
||||
json={
|
||||
"workspace_id": WORKSPACE_ID,
|
||||
"tool_names": "web_search"
|
||||
}
|
||||
)
|
||||
|
||||
result = response.json()
|
||||
if result.get("success"):
|
||||
memory_list = result.get("metadata", {}).get("memory_list", [])
|
||||
for memory in memory_list:
|
||||
print(f"Tool: {memory['when_to_use']}")
|
||||
print(f"Guidelines:\n{memory['content']}")
|
||||
print(f"Total Calls: {len(memory['tool_call_results'])}")
|
||||
```
|
||||
|
||||
#### Multiple Tools Retrieval
|
||||
|
||||
```python
|
||||
# Retrieve memories for multiple tools at once
|
||||
response = requests.post(
|
||||
url=f"{BASE_URL}retrieve_tool_memory",
|
||||
json={
|
||||
"workspace_id": WORKSPACE_ID,
|
||||
"tool_names": "web_search,file_reader,api_caller,database_query"
|
||||
}
|
||||
)
|
||||
|
||||
result = response.json()
|
||||
if result.get("success"):
|
||||
memory_list = result.get("metadata", {}).get("memory_list", [])
|
||||
print(f"Retrieved {len(memory_list)} tool memories")
|
||||
|
||||
for memory in memory_list:
|
||||
print(f"\n{'='*60}")
|
||||
print(f"Tool: {memory['when_to_use']}")
|
||||
print(f"{'='*60}")
|
||||
print(memory['content'])
|
||||
```
|
||||
|
||||
#### Extracting Specific Information
|
||||
|
||||
```python
|
||||
def get_tool_statistics(tool_name):
|
||||
"""Get statistical information for a specific tool"""
|
||||
response = requests.post(
|
||||
url=f"{BASE_URL}retrieve_tool_memory",
|
||||
json={
|
||||
"workspace_id": WORKSPACE_ID,
|
||||
"tool_names": tool_name
|
||||
}
|
||||
)
|
||||
|
||||
result = response.json()
|
||||
if not result.get("success"):
|
||||
return None
|
||||
|
||||
memory_list = result.get("metadata", {}).get("memory_list", [])
|
||||
if not memory_list:
|
||||
return None
|
||||
|
||||
memory = memory_list[0]
|
||||
tool_calls = memory['tool_call_results']
|
||||
|
||||
# Calculate statistics
|
||||
total_calls = len(tool_calls)
|
||||
successful_calls = sum(1 for call in tool_calls if call['success'])
|
||||
avg_score = sum(call['score'] for call in tool_calls) / total_calls if total_calls > 0 else 0
|
||||
avg_time = sum(call['time_cost'] for call in tool_calls) / total_calls if total_calls > 0 else 0
|
||||
avg_tokens = sum(call['token_cost'] for call in tool_calls) / total_calls if total_calls > 0 else 0
|
||||
|
||||
return {
|
||||
"tool_name": tool_name,
|
||||
"total_calls": total_calls,
|
||||
"success_rate": successful_calls / total_calls if total_calls > 0 else 0,
|
||||
"avg_score": avg_score,
|
||||
"avg_time_cost": avg_time,
|
||||
"avg_token_cost": avg_tokens,
|
||||
"guidelines": memory['content']
|
||||
}
|
||||
|
||||
# Usage
|
||||
stats = get_tool_statistics("web_search")
|
||||
if stats:
|
||||
print(f"Tool: {stats['tool_name']}")
|
||||
print(f"Total Calls: {stats['total_calls']}")
|
||||
print(f"Success Rate: {stats['success_rate']:.1%}")
|
||||
print(f"Avg Score: {stats['avg_score']:.2f}")
|
||||
print(f"Avg Time: {stats['avg_time_cost']:.2f}s")
|
||||
print(f"Avg Tokens: {stats['avg_token_cost']:.0f}")
|
||||
```
|
||||
|
||||
### Integration with Agent Workflows
|
||||
|
||||
#### Pre-Execution Retrieval
|
||||
|
||||
```python
|
||||
def execute_tool_with_guidelines(tool_name, input_params):
|
||||
"""Execute a tool after retrieving its usage guidelines"""
|
||||
|
||||
# 1. Retrieve tool memory
|
||||
response = requests.post(
|
||||
url=f"{BASE_URL}retrieve_tool_memory",
|
||||
json={
|
||||
"workspace_id": WORKSPACE_ID,
|
||||
"tool_names": tool_name
|
||||
}
|
||||
)
|
||||
|
||||
# 2. Extract guidelines
|
||||
guidelines = ""
|
||||
if response.json().get("success"):
|
||||
memory_list = response.json().get("metadata", {}).get("memory_list", [])
|
||||
if memory_list:
|
||||
guidelines = memory_list[0]['content']
|
||||
print(f"Guidelines for {tool_name}:")
|
||||
print(guidelines)
|
||||
|
||||
# 3. Adjust parameters based on guidelines
|
||||
# (This would be done by an LLM or rule-based system)
|
||||
adjusted_params = adjust_parameters(input_params, guidelines)
|
||||
|
||||
# 4. Execute tool
|
||||
result = execute_tool(tool_name, adjusted_params)
|
||||
|
||||
# 5. Record result
|
||||
record_tool_call(tool_name, adjusted_params, result)
|
||||
|
||||
return result
|
||||
```
|
||||
|
||||
#### Batch Retrieval for Agent Initialization
|
||||
|
||||
```python
|
||||
def initialize_agent_with_tool_memories(available_tools):
|
||||
"""Load all tool memories at agent initialization"""
|
||||
|
||||
# Retrieve all tool memories at once
|
||||
tool_names = ",".join(available_tools)
|
||||
response = requests.post(
|
||||
url=f"{BASE_URL}retrieve_tool_memory",
|
||||
json={
|
||||
"workspace_id": WORKSPACE_ID,
|
||||
"tool_names": tool_names
|
||||
}
|
||||
)
|
||||
|
||||
# Build tool memory cache
|
||||
tool_memory_cache = {}
|
||||
if response.json().get("success"):
|
||||
memory_list = response.json().get("metadata", {}).get("memory_list", [])
|
||||
for memory in memory_list:
|
||||
tool_memory_cache[memory['when_to_use']] = {
|
||||
"guidelines": memory['content'],
|
||||
"total_calls": len(memory['tool_call_results']),
|
||||
"last_modified": memory['time_modified']
|
||||
}
|
||||
|
||||
return tool_memory_cache
|
||||
|
||||
# Usage
|
||||
available_tools = ["web_search", "file_reader", "api_caller"]
|
||||
tool_cache = initialize_agent_with_tool_memories(available_tools)
|
||||
|
||||
# Agent can now quickly access guidelines
|
||||
if "web_search" in tool_cache:
|
||||
print(tool_cache["web_search"]["guidelines"])
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
|
||||
#### Tool Not Found
|
||||
|
||||
```python
|
||||
response = requests.post(
|
||||
url=f"{BASE_URL}retrieve_tool_memory",
|
||||
json={
|
||||
"workspace_id": WORKSPACE_ID,
|
||||
"tool_names": "nonexistent_tool"
|
||||
}
|
||||
)
|
||||
|
||||
result = response.json()
|
||||
if not result.get("success"):
|
||||
print(f"Error: {result.get('answer')}")
|
||||
# Output: "No matching tool memories found"
|
||||
```
|
||||
|
||||
#### Empty Tool Names
|
||||
|
||||
```python
|
||||
response = requests.post(
|
||||
url=f"{BASE_URL}retrieve_tool_memory",
|
||||
json={
|
||||
"workspace_id": WORKSPACE_ID,
|
||||
"tool_names": ""
|
||||
}
|
||||
)
|
||||
|
||||
result = response.json()
|
||||
if not result.get("success"):
|
||||
print(f"Error: {result.get('answer')}")
|
||||
# Output: "tool_names is required"
|
||||
```
|
||||
|
||||
#### Partial Matches
|
||||
|
||||
```python
|
||||
# Request 3 tools, but only 2 exist
|
||||
response = requests.post(
|
||||
url=f"{BASE_URL}retrieve_tool_memory",
|
||||
json={
|
||||
"workspace_id": WORKSPACE_ID,
|
||||
"tool_names": "web_search,file_reader,nonexistent_tool"
|
||||
}
|
||||
)
|
||||
|
||||
result = response.json()
|
||||
if result.get("success"):
|
||||
memory_list = result.get("metadata", {}).get("memory_list", [])
|
||||
print(f"Found {len(memory_list)} out of 3 requested tools")
|
||||
# Output: "Found 2 out of 3 requested tools"
|
||||
```
|
||||
|
||||
### Retrieval Workflow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Agent
|
||||
participant RetrieveOp
|
||||
participant VectorStore
|
||||
|
||||
Agent->>RetrieveOp: retrieve_tool_memory(tool_names)
|
||||
RetrieveOp->>RetrieveOp: Split and validate names
|
||||
|
||||
loop For each tool name
|
||||
RetrieveOp->>VectorStore: search(tool_name, top_k=1)
|
||||
VectorStore-->>RetrieveOp: Return top result
|
||||
RetrieveOp->>RetrieveOp: Validate type and name
|
||||
alt Valid match
|
||||
RetrieveOp->>RetrieveOp: Add to results
|
||||
else No match
|
||||
RetrieveOp->>RetrieveOp: Log warning
|
||||
end
|
||||
end
|
||||
|
||||
RetrieveOp-->>Agent: Return memory_list
|
||||
Agent->>Agent: Apply guidelines
|
||||
```
|
||||
|
||||
### Use Cases
|
||||
|
||||
#### Use Case 1: Pre-Execution Guidance
|
||||
|
||||
**Scenario**: Before executing a tool, the agent retrieves usage guidelines to optimize parameters.
|
||||
|
||||
```python
|
||||
# Agent needs to search the web
|
||||
tool_name = "web_search"
|
||||
query = "machine learning basics"
|
||||
|
||||
# Retrieve guidelines
|
||||
response = requests.post(
|
||||
url=f"{BASE_URL}retrieve_tool_memory",
|
||||
json={
|
||||
"workspace_id": WORKSPACE_ID,
|
||||
"tool_names": tool_name
|
||||
}
|
||||
)
|
||||
|
||||
# Guidelines suggest: max_results=10-20, use filter_type="technical_docs"
|
||||
# Agent adjusts parameters accordingly
|
||||
optimized_params = {
|
||||
"query": query,
|
||||
"max_results": 15, # Within recommended range
|
||||
"filter_type": "technical_docs", # As suggested
|
||||
"language": "en"
|
||||
}
|
||||
|
||||
# Execute with optimized parameters
|
||||
result = execute_tool(tool_name, optimized_params)
|
||||
```
|
||||
|
||||
#### Use Case 2: Performance Monitoring
|
||||
|
||||
**Scenario**: Monitor tool performance trends over time.
|
||||
|
||||
```python
|
||||
def monitor_tool_performance(tool_name):
|
||||
"""Monitor tool performance and detect degradation"""
|
||||
response = requests.post(
|
||||
url=f"{BASE_URL}retrieve_tool_memory",
|
||||
json={
|
||||
"workspace_id": WORKSPACE_ID,
|
||||
"tool_names": tool_name
|
||||
}
|
||||
)
|
||||
|
||||
if not response.json().get("success"):
|
||||
return None
|
||||
|
||||
memory = response.json()["metadata"]["memory_list"][0]
|
||||
calls = memory["tool_call_results"]
|
||||
|
||||
# Analyze recent vs historical performance
|
||||
recent_calls = calls[-20:] # Last 20 calls
|
||||
historical_calls = calls[:-20] if len(calls) > 20 else []
|
||||
|
||||
recent_success_rate = sum(1 for c in recent_calls if c['success']) / len(recent_calls)
|
||||
historical_success_rate = (sum(1 for c in historical_calls if c['success']) / len(historical_calls)
|
||||
if historical_calls else recent_success_rate)
|
||||
|
||||
# Detect degradation
|
||||
if recent_success_rate < historical_success_rate - 0.1:
|
||||
print(f"Warning: {tool_name} performance degraded!")
|
||||
print(f"Recent: {recent_success_rate:.1%}, Historical: {historical_success_rate:.1%}")
|
||||
return "degraded"
|
||||
|
||||
return "healthy"
|
||||
|
||||
# Usage
|
||||
status = monitor_tool_performance("web_search")
|
||||
```
|
||||
|
||||
#### Use Case 3: Tool Selection
|
||||
|
||||
**Scenario**: Choose the best tool for a task based on historical performance.
|
||||
|
||||
```python
|
||||
def select_best_tool(task_description, candidate_tools):
|
||||
"""Select the best tool based on historical performance"""
|
||||
|
||||
# Retrieve memories for all candidate tools
|
||||
tool_names = ",".join(candidate_tools)
|
||||
response = requests.post(
|
||||
url=f"{BASE_URL}retrieve_tool_memory",
|
||||
json={
|
||||
"workspace_id": WORKSPACE_ID,
|
||||
"tool_names": tool_names
|
||||
}
|
||||
)
|
||||
|
||||
if not response.json().get("success"):
|
||||
return candidate_tools[0] # Default to first
|
||||
|
||||
memory_list = response.json()["metadata"]["memory_list"]
|
||||
|
||||
# Score each tool
|
||||
tool_scores = {}
|
||||
for memory in memory_list:
|
||||
calls = memory["tool_call_results"]
|
||||
if not calls:
|
||||
continue
|
||||
|
||||
# Calculate composite score
|
||||
success_rate = sum(1 for c in calls if c['success']) / len(calls)
|
||||
avg_score = sum(c['score'] for c in calls) / len(calls)
|
||||
avg_time = sum(c['time_cost'] for c in calls) / len(calls)
|
||||
|
||||
# Composite: prioritize success and quality, penalize slow tools
|
||||
composite = (success_rate * 0.4 + avg_score * 0.4 + (1 / (1 + avg_time)) * 0.2)
|
||||
tool_scores[memory['when_to_use']] = composite
|
||||
|
||||
# Select best tool
|
||||
if tool_scores:
|
||||
best_tool = max(tool_scores, key=tool_scores.get)
|
||||
print(f"Selected {best_tool} with score {tool_scores[best_tool]:.2f}")
|
||||
return best_tool
|
||||
|
||||
return candidate_tools[0]
|
||||
|
||||
# Usage
|
||||
best = select_best_tool(
|
||||
"Search for technical documentation",
|
||||
["web_search", "doc_search", "api_search"]
|
||||
)
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Retrieval Timing**:
|
||||
- Retrieve guidelines before first use of a tool
|
||||
- Cache retrieved memories for the duration of a task
|
||||
- Re-retrieve after tool memory updates (post-summarization)
|
||||
|
||||
2. **Batch Retrieval**:
|
||||
- Retrieve multiple tool memories in a single request
|
||||
- Use comma-separated tool names for efficiency
|
||||
- Initialize agent with all available tool memories
|
||||
|
||||
3. **Error Handling**:
|
||||
- Always check `success` status in response
|
||||
- Handle cases where tool memory doesn't exist
|
||||
- Provide fallback behavior for missing guidelines
|
||||
|
||||
4. **Guidelines Application**:
|
||||
- Parse guidelines to extract parameter recommendations
|
||||
- Use LLM to interpret guidelines in context
|
||||
- Combine guidelines with task-specific requirements
|
||||
|
||||
5. **Performance Optimization**:
|
||||
- Cache retrieved memories to avoid repeated calls
|
||||
- Invalidate cache after summarization updates
|
||||
- Monitor retrieval latency and optimize if needed
|
||||
|
||||
6. **Monitoring**:
|
||||
- Log retrieval requests for debugging
|
||||
- Track which tools are frequently retrieved
|
||||
- Identify tools without memory (candidates for recording)
|
||||
|
||||
|
|
|
|||
|
|
@ -10,202 +10,19 @@ Evaluates individual tool invocations and adds them to the tool memory database
|
|||
|
||||
- Receives tool call results with input parameters, output, and metadata
|
||||
- Uses LLM to evaluate each tool call based on success and parameter alignment
|
||||
- Generates summary, evaluation, and score (0.0, 0.5, or 1.0) for each call
|
||||
- Generates summary, evaluation, and score (0.0 or 1.0) for each call
|
||||
- Appends evaluated results to existing tool memory or creates new memory
|
||||
- Maintains a sliding window of recent tool calls (configurable limit)
|
||||
|
||||
### Processing Flow
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
A[Receive Tool Call Results] --> B[Validate Input]
|
||||
B --> C[Search for Existing Memory]
|
||||
C --> D{Memory Exists?}
|
||||
D -->|Yes| E[Load Existing Memory]
|
||||
D -->|No| F[Create New Memory]
|
||||
E --> G[Concurrent Evaluation]
|
||||
F --> G
|
||||
G --> H[Evaluate Call #1]
|
||||
G --> I[Evaluate Call #2]
|
||||
G --> J[Evaluate Call #N]
|
||||
H --> K[Generate Summary & Score]
|
||||
I --> K
|
||||
J --> K
|
||||
K --> L[Append to Memory]
|
||||
L --> M[Trim to Max History]
|
||||
M --> N[Update Modified Time]
|
||||
N --> O[Prepare for Vector Store]
|
||||
```
|
||||
|
||||
1. **Input Validation**:
|
||||
- Verify `tool_name` is provided
|
||||
- Convert dict objects to `ToolCallResult` instances
|
||||
- Check if `tool_call_results` list is not empty
|
||||
|
||||
2. **Memory Lookup**:
|
||||
- Search vector store for existing tool memory by tool name
|
||||
- Verify exact match (memory type = "tool" and when_to_use = tool_name)
|
||||
- Create new `ToolMemory` if no match found
|
||||
|
||||
3. **Concurrent Evaluation**:
|
||||
- Submit all tool call results for parallel evaluation
|
||||
- Each evaluation uses LLM to analyze the call
|
||||
- Generate structured evaluation with summary, assessment, and score
|
||||
|
||||
4. **Memory Update**:
|
||||
- Append evaluated results to tool memory
|
||||
- Trim to `max_history_tool_call_cnt` if limit exceeded
|
||||
- Update modification timestamp
|
||||
- Prepare for vector store update
|
||||
|
||||
### Parameters
|
||||
|
||||
Configure in `default.yaml`:
|
||||
|
||||
```yaml
|
||||
op:
|
||||
parse_tool_call_result_op:
|
||||
backend: parse_tool_call_result_op
|
||||
llm: default
|
||||
params:
|
||||
max_history_tool_call_cnt: 100
|
||||
evaluation_sleep_interval: 1.0
|
||||
```
|
||||
|
||||
- `max_history_tool_call_cnt` (integer, default: `100`):
|
||||
- `op.parse_tool_call_result_op.params.max_history_tool_call_cnt` (integer, default: `100`):
|
||||
- Maximum number of historical tool call results to retain per tool
|
||||
- When exceeded, oldest results are removed (FIFO)
|
||||
- Balances memory size with historical context
|
||||
- Recommended: 50-200 depending on tool usage frequency
|
||||
|
||||
- `evaluation_sleep_interval` (float, default: `1.0`):
|
||||
- `op.parse_tool_call_result_op.params.evaluation_sleep_interval` (float, default: `1.0`):
|
||||
- Delay in seconds between concurrent evaluations
|
||||
- Prevents rate limiting when evaluating multiple calls
|
||||
- Set to 0 for maximum speed (if no rate limits)
|
||||
- Increase if encountering API throttling
|
||||
|
||||
### Evaluation Criteria
|
||||
|
||||
The LLM evaluates each tool call based on two dimensions:
|
||||
|
||||
1. **Success Evaluation**:
|
||||
- Check if the success flag indicates successful execution
|
||||
- Verify output contains no error messages
|
||||
- Assess if time and token costs are reasonable
|
||||
- Identify any error indicators in the output
|
||||
|
||||
2. **Parameter Alignment Evaluation**:
|
||||
- Evaluate if output matches expected behavior given input
|
||||
- Consider if input parameters are appropriate for the tool
|
||||
- Check for parameter mismatches or unexpected behaviors
|
||||
- Verify output is consistent with tool's intended purpose
|
||||
|
||||
### Scoring Guidelines
|
||||
|
||||
- **1.0 (Success)**: Tool executed successfully with good parameter alignment
|
||||
- Example: Query returned relevant results, parameters were appropriate
|
||||
|
||||
- **0.5 (Partial Success)**: Tool executed but with issues
|
||||
- Example: Query succeeded but parameters were suboptimal (e.g., too generic)
|
||||
- Example: Results returned but with warnings about invalid parameters
|
||||
|
||||
- **0.0 (Failure)**: Tool execution failed or severe parameter misalignment
|
||||
- Example: Timeout due to excessive max_results parameter
|
||||
- Example: Error due to invalid parameter format
|
||||
|
||||
### Output Format
|
||||
|
||||
The operation sets the following in `context.response.metadata`:
|
||||
|
||||
```python
|
||||
{
|
||||
"deleted_memory_ids": ["memory_id_if_updating"],
|
||||
"memory_list": [
|
||||
{
|
||||
"memory_id": "abc123",
|
||||
"when_to_use": "web_search",
|
||||
"tool_call_results": [
|
||||
{
|
||||
"create_time": "2025-10-15 14:30:00",
|
||||
"tool_name": "web_search",
|
||||
"input": {...},
|
||||
"output": "...",
|
||||
"summary": "Successfully retrieved 10 relevant results",
|
||||
"evaluation": "Good parameter alignment...",
|
||||
"score": 1.0,
|
||||
"success": True,
|
||||
"time_cost": 2.3,
|
||||
"token_cost": 150
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Usage Example
|
||||
|
||||
```python
|
||||
import requests
|
||||
from datetime import datetime
|
||||
|
||||
BASE_URL = "http://0.0.0.0:8002/"
|
||||
WORKSPACE_ID = "demo_workspace"
|
||||
|
||||
# Prepare tool call results
|
||||
tool_call_results = [
|
||||
{
|
||||
"create_time": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"tool_name": "web_search",
|
||||
"input": {
|
||||
"query": "Python asyncio tutorial",
|
||||
"max_results": 10,
|
||||
"language": "en",
|
||||
"filter_type": "technical_docs"
|
||||
},
|
||||
"output": "Found 10 relevant results including official documentation and tutorials",
|
||||
"token_cost": 150,
|
||||
"success": True,
|
||||
"time_cost": 2.3
|
||||
},
|
||||
{
|
||||
"create_time": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"tool_name": "web_search",
|
||||
"input": {
|
||||
"query": "test", # Too generic
|
||||
"max_results": 100, # Too many
|
||||
"language": "unknown" # Invalid
|
||||
},
|
||||
"output": "Warning: language 'unknown' not supported. Query too generic, limited results.",
|
||||
"token_cost": 80,
|
||||
"success": True,
|
||||
"time_cost": 3.5
|
||||
}
|
||||
]
|
||||
|
||||
# Add tool call results
|
||||
response = requests.post(
|
||||
url=f"{BASE_URL}add_tool_call_result",
|
||||
json={
|
||||
"workspace_id": WORKSPACE_ID,
|
||||
"tool_name": "web_search",
|
||||
"tool_call_results": tool_call_results
|
||||
}
|
||||
)
|
||||
|
||||
result = response.json()
|
||||
print(f"Success: {result.get('success')}")
|
||||
print(f"Answer: {result.get('answer')}")
|
||||
|
||||
# Check evaluated results
|
||||
memory_list = result.get("metadata", {}).get("memory_list", [])
|
||||
if memory_list:
|
||||
tool_memory = memory_list[0]
|
||||
for call_result in tool_memory["tool_call_results"]:
|
||||
print(f"\nCall Summary: {call_result['summary']}")
|
||||
print(f"Evaluation: {call_result['evaluation']}")
|
||||
print(f"Score: {call_result['score']}")
|
||||
```
|
||||
|
||||
## SummaryToolMemoryOp
|
||||
|
||||
|
|
@ -221,245 +38,13 @@ Analyzes accumulated tool call history and generates comprehensive usage pattern
|
|||
- Uses LLM to synthesize actionable usage guidelines
|
||||
- Updates tool memory content with generated insights
|
||||
|
||||
### Processing Flow
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
A[Receive Tool Names] --> B[Split by Comma]
|
||||
B --> C[For Each Tool Name]
|
||||
C --> D[Search Vector Store]
|
||||
D --> E{Exact Match?}
|
||||
E -->|Yes| F[Load Tool Memory]
|
||||
E -->|No| G[Log Warning]
|
||||
F --> H[Extract Recent N Calls]
|
||||
H --> I[Calculate Statistics]
|
||||
I --> J[Format Call Summaries]
|
||||
J --> K[Format Statistics]
|
||||
K --> L[Concurrent Summarization]
|
||||
L --> M[LLM Analysis #1]
|
||||
L --> N[LLM Analysis #2]
|
||||
L --> O[LLM Analysis #N]
|
||||
M --> P[Generate Guidelines]
|
||||
N --> P
|
||||
O --> P
|
||||
P --> Q[Update Memory Content]
|
||||
Q --> R[Update Modified Time]
|
||||
R --> S[Prepare for Vector Store]
|
||||
```
|
||||
|
||||
1. **Tool Name Processing**:
|
||||
- Split comma-separated tool names
|
||||
- Trim whitespace from each name
|
||||
- Log the list of tools to process
|
||||
|
||||
2. **Memory Retrieval**:
|
||||
- Search vector store for each tool name
|
||||
- Verify exact match (memory type and when_to_use)
|
||||
- Skip tools without existing memory
|
||||
|
||||
3. **Data Preparation**:
|
||||
- Extract the most recent N tool call results
|
||||
- Calculate statistical metrics:
|
||||
- Total calls vs recent calls analyzed
|
||||
- Success rate (overall and recent)
|
||||
- Average score (overall and recent)
|
||||
- Average time cost
|
||||
- Average token cost
|
||||
- Format call summaries as markdown
|
||||
- Format statistics as markdown
|
||||
|
||||
4. **Concurrent Summarization**:
|
||||
- Submit all tools for parallel summarization
|
||||
- Each summarization uses LLM to analyze patterns
|
||||
- Generate structured usage guidelines
|
||||
|
||||
5. **Memory Update**:
|
||||
- Update tool memory content with new guidelines
|
||||
- Update modification timestamp
|
||||
- Prepare for vector store update
|
||||
|
||||
### Parameters
|
||||
|
||||
Configure in `default.yaml`:
|
||||
|
||||
```yaml
|
||||
op:
|
||||
summary_tool_memory_op:
|
||||
backend: summary_tool_memory_op
|
||||
llm: default
|
||||
params:
|
||||
recent_call_count: 20
|
||||
summary_sleep_interval: 1.0
|
||||
```
|
||||
|
||||
- `recent_call_count` (integer, default: `20`):
|
||||
- `op.summary_tool_memory_op.params.recent_call_count` (integer, default: `30`):
|
||||
- Number of most recent tool calls to analyze
|
||||
- Focuses on recent usage patterns
|
||||
- Recommended: 10-50 depending on tool usage frequency
|
||||
- Higher values provide more context but may dilute recent patterns
|
||||
|
||||
- `summary_sleep_interval` (float, default: `1.0`):
|
||||
- `op.summary_tool_memory_op.params.summary_sleep_interval` (float, default: `1.0`):
|
||||
- Delay in seconds between concurrent summarizations
|
||||
- Prevents rate limiting when summarizing multiple tools
|
||||
- Set to 0 for maximum speed (if no rate limits)
|
||||
- Increase if encountering API throttling
|
||||
|
||||
### Statistical Metrics
|
||||
|
||||
The operation calculates the following metrics:
|
||||
|
||||
```python
|
||||
{
|
||||
"total_calls": 100, # Total number of calls in history
|
||||
"recent_calls": 20, # Number of recent calls analyzed
|
||||
"success_rate": 0.85, # Overall success rate (85%)
|
||||
"recent_success_rate": 0.90, # Recent success rate (90%)
|
||||
"avg_score": 0.78, # Average evaluation score
|
||||
"recent_avg_score": 0.82, # Recent average score
|
||||
"avg_time_cost": 2.45, # Average time in seconds
|
||||
"avg_token_cost": 125.3 # Average token consumption
|
||||
}
|
||||
```
|
||||
|
||||
### Generated Guidelines Structure
|
||||
|
||||
The LLM generates guidelines following this structure:
|
||||
|
||||
1. **Core Function**: What the tool does and when to use it
|
||||
2. **Success Patterns**: Parameter patterns and scenarios that work well
|
||||
3. **Common Issues**: Main pitfalls to avoid and why they fail
|
||||
4. **Best Practices**: 2-3 actionable recommendations
|
||||
|
||||
### Usage Example
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
BASE_URL = "http://0.0.0.0:8002/"
|
||||
WORKSPACE_ID = "demo_workspace"
|
||||
|
||||
# After accumulating tool call history, generate guidelines
|
||||
response = requests.post(
|
||||
url=f"{BASE_URL}summary_tool_memory",
|
||||
json={
|
||||
"workspace_id": WORKSPACE_ID,
|
||||
"tool_names": "web_search,file_reader,api_caller" # Multiple tools
|
||||
}
|
||||
)
|
||||
|
||||
result = response.json()
|
||||
print(f"Success: {result.get('success')}")
|
||||
print(f"Answer: {result.get('answer')}")
|
||||
|
||||
# Display generated guidelines
|
||||
memory_list = result.get("metadata", {}).get("memory_list", [])
|
||||
for memory in memory_list:
|
||||
print(f"\n{'='*60}")
|
||||
print(f"Tool: {memory['when_to_use']}")
|
||||
print(f"{'='*60}")
|
||||
print(memory['content'])
|
||||
|
||||
# Display statistics
|
||||
stats = memory.get('metadata', {}).get('statistics', {})
|
||||
print(f"\nStatistics:")
|
||||
print(f" Total Calls: {stats.get('total_calls', 0)}")
|
||||
print(f" Success Rate: {stats.get('success_rate', 0):.1%}")
|
||||
print(f" Avg Score: {stats.get('avg_score', 0):.2f}")
|
||||
```
|
||||
|
||||
### Example Generated Guidelines
|
||||
|
||||
```
|
||||
Core Function:
|
||||
The web_search tool retrieves information from the internet based on query parameters.
|
||||
Use it when you need up-to-date information, documentation, or external data.
|
||||
|
||||
Success Patterns:
|
||||
- Specific queries (e.g., "Python asyncio tutorial") achieve 95% success rate
|
||||
- Setting max_results=10-20 balances quality and performance
|
||||
- Using language="en" and filter_type="technical_docs" improves relevance
|
||||
- Average successful call: 2.3s, 150 tokens
|
||||
|
||||
Common Issues:
|
||||
- Generic queries (e.g., "test") return poor results (score: 0.5)
|
||||
- max_results > 50 often leads to timeouts (avg: 8.2s vs 2.3s)
|
||||
- Invalid language codes default to English but add latency
|
||||
- Missing filter_type returns mixed-quality results
|
||||
|
||||
Best Practices:
|
||||
1. Use specific, descriptive queries with clear intent
|
||||
2. Set max_results=10-20 for optimal balance
|
||||
3. Always specify language and filter_type for technical searches
|
||||
```
|
||||
|
||||
### When to Run Summarization
|
||||
|
||||
- **Initial Setup**: After accumulating 20-30 tool calls
|
||||
- **Regular Updates**: Every 50-100 new calls
|
||||
- **Pattern Changes**: When success rate changes significantly
|
||||
- **Tool Updates**: After tool version changes or parameter updates
|
||||
- **Performance Issues**: When investigating declining performance
|
||||
|
||||
### Integration with Retrieval
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Agent
|
||||
participant AddOp as add_tool_call_result
|
||||
participant SummaryOp as summary_tool_memory
|
||||
participant RetrieveOp as retrieve_tool_memory
|
||||
|
||||
loop Every Tool Call
|
||||
Agent->>AddOp: Record tool call result
|
||||
AddOp->>AddOp: Evaluate and store
|
||||
end
|
||||
|
||||
Note over Agent,SummaryOp: After 20+ calls
|
||||
Agent->>SummaryOp: Generate guidelines
|
||||
SummaryOp->>SummaryOp: Analyze patterns
|
||||
SummaryOp->>SummaryOp: Update content
|
||||
|
||||
Note over Agent,RetrieveOp: Before next use
|
||||
Agent->>RetrieveOp: Get tool memory
|
||||
RetrieveOp-->>Agent: Return guidelines + history
|
||||
Agent->>Agent: Apply recommendations
|
||||
```
|
||||
|
||||
The summarization operation works in conjunction with retrieval:
|
||||
1. `add_tool_call_result` continuously records invocations
|
||||
2. `summary_tool_memory` periodically generates guidelines
|
||||
3. `retrieve_tool_memory` provides guidelines before tool use
|
||||
4. Agent applies recommendations to improve success rates
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Recording Strategy**:
|
||||
- Record every tool invocation, including failures
|
||||
- Include detailed input parameters and complete output
|
||||
- Capture accurate performance metrics
|
||||
- Add relevant metadata for context
|
||||
|
||||
2. **Evaluation Quality**:
|
||||
- Ensure LLM has sufficient context for evaluation
|
||||
- Monitor score distribution (should not be all 1.0 or 0.0)
|
||||
- Review evaluations periodically for quality
|
||||
- Adjust evaluation prompts if needed
|
||||
|
||||
3. **Summarization Timing**:
|
||||
- Wait for 20-30 calls before first summarization
|
||||
- Re-summarize after significant new data (50+ calls)
|
||||
- Update when usage patterns change
|
||||
- Regenerate after tool updates
|
||||
|
||||
4. **Parameter Tuning**:
|
||||
- Adjust `max_history_tool_call_cnt` based on tool usage frequency
|
||||
- Increase `recent_call_count` for tools with diverse patterns
|
||||
- Balance `sleep_interval` between speed and rate limits
|
||||
- Monitor vector store size and adjust retention limits
|
||||
|
||||
5. **Quality Maintenance**:
|
||||
- Review generated guidelines for accuracy
|
||||
- Validate statistical metrics match expectations
|
||||
- Clean up deprecated tools from vector store
|
||||
- Archive historical data before major changes
|
||||
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||
|
||||
[project]
|
||||
name = "reme_ai"
|
||||
version = "0.1.9"
|
||||
version = "0.1.10"
|
||||
description = "Remember me"
|
||||
authors = [
|
||||
{ name = "jinli.yl", email = "jinli.yl@alibaba-inc.com" },
|
||||
|
|
@ -24,7 +24,7 @@ classifiers = [
|
|||
keywords = ["llm", "memory", "experience", "memoryscope", "ai", "mcp", "http"]
|
||||
|
||||
dependencies = [
|
||||
"flowllm==0.1.9",
|
||||
"flowllm>=0.1.11.1",
|
||||
]
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
|
|
|
|||
|
|
@ -1,6 +1,14 @@
|
|||
import warnings
|
||||
|
||||
from pydantic.warnings import PydanticDeprecatedSince20
|
||||
|
||||
warnings.filterwarnings("ignore", category=DeprecationWarning, module="websockets")
|
||||
warnings.filterwarnings("ignore", category=DeprecationWarning, module="uvicorn")
|
||||
warnings.filterwarnings("ignore", category=PydanticDeprecatedSince20)
|
||||
|
||||
from . import agent
|
||||
from . import retrieve
|
||||
from . import summary
|
||||
from . import vector_store
|
||||
|
||||
__version__ = "0.1.9"
|
||||
__version__ = "0.1.10"
|
||||
|
|
|
|||
|
|
@ -34,6 +34,7 @@ class LLMMockSearchOp(BaseAsyncToolOp):
|
|||
simple_config: Dict[str, Any] = None,
|
||||
medium_config: Dict[str, Any] = None,
|
||||
complex_config: Dict[str, Any] = None,
|
||||
seed: int = 0,
|
||||
**kwargs):
|
||||
"""
|
||||
Initialize the LLM Mock Search Op.
|
||||
|
|
@ -52,8 +53,13 @@ class LLMMockSearchOp(BaseAsyncToolOp):
|
|||
- success_rate: float (0-1), default 0.70
|
||||
- extra_time: float (seconds), default 1.5
|
||||
- relevance_ratio: float (0-1), default 0.80
|
||||
seed: Random seed for deterministic behavior, default 0
|
||||
"""
|
||||
super().__init__(llm=llm, **kwargs)
|
||||
|
||||
# Set random seed for deterministic behavior
|
||||
self.seed = seed
|
||||
random.seed(self.seed)
|
||||
|
||||
# Default configurations for each scenario
|
||||
self.simple_config = {
|
||||
|
|
@ -221,13 +227,16 @@ class LLMMockSearchOp(BaseAsyncToolOp):
|
|||
# Step 5: Check relevance ratio
|
||||
if random.random() > config["relevance_ratio"]:
|
||||
# Generate random/irrelevant result
|
||||
logger.info("Generating low relevance result")
|
||||
# NOTE: success=True because technically the tool executed without errors,
|
||||
# but the content is irrelevant (low quality), which should result in score=0.0 during evaluation
|
||||
logger.info("Generating low relevance result (success=True but low quality)")
|
||||
content = await self.generate_random_result()
|
||||
result_dict = {
|
||||
"success": False,
|
||||
"success": True, # Technical execution succeeded
|
||||
"content": content,
|
||||
"query": query,
|
||||
"complexity": complexity
|
||||
"complexity": complexity,
|
||||
"is_relevant": False # Mark as irrelevant for debugging
|
||||
}
|
||||
else:
|
||||
# Generate relevant result
|
||||
|
|
@ -237,7 +246,8 @@ class LLMMockSearchOp(BaseAsyncToolOp):
|
|||
"success": True,
|
||||
"content": content,
|
||||
"query": query,
|
||||
"complexity": complexity
|
||||
"complexity": complexity,
|
||||
"is_relevant": True # Mark as relevant for debugging
|
||||
}
|
||||
|
||||
self.set_result(json.dumps(result_dict, ensure_ascii=False))
|
||||
|
|
|
|||
|
|
@ -9,23 +9,23 @@ class SearchToolA(LLMMockSearchOp):
|
|||
def __init__(self, llm: str = "qwen3_30b_instruct", **kwargs):
|
||||
# Configure for fast but shallow performance
|
||||
simple_config = {
|
||||
"success_rate": 0.95, # High success rate for simple queries
|
||||
"extra_time": 0.2, # Very fast (0.2-0.5s range)
|
||||
"relevance_ratio": 0.92, # High relevance
|
||||
"success_rate": 0.9, # High success rate for simple queries
|
||||
"extra_time": 0, # Very fast (0.2-0.5s range)
|
||||
"relevance_ratio": 0.9, # High relevance
|
||||
"content_length": "short" # Concise answers
|
||||
}
|
||||
|
||||
medium_config = {
|
||||
"success_rate": 0.75, # Lower success for medium queries
|
||||
"extra_time": 0.3, # Still fast
|
||||
"relevance_ratio": 0.70, # Moderate relevance
|
||||
"success_rate": 0.2, # Lower success for medium queries
|
||||
"extra_time": 0, # Still fast
|
||||
"relevance_ratio": 0.2, # Moderate relevance
|
||||
"content_length": "short" # Limited depth
|
||||
}
|
||||
|
||||
complex_config = {
|
||||
"success_rate": 0.50, # Poor success rate for complex queries
|
||||
"extra_time": 0.4, # Fast but insufficient
|
||||
"relevance_ratio": 0.50, # Low relevance (often misses key aspects)
|
||||
"success_rate": 0.5, # Poor success rate for complex queries
|
||||
"extra_time": 0, # Fast but insufficient
|
||||
"relevance_ratio": 0.5, # Low relevance (often misses key aspects)
|
||||
"content_length": "short" # Too shallow for complex topics
|
||||
}
|
||||
|
||||
|
|
@ -45,23 +45,23 @@ class SearchToolB(LLMMockSearchOp):
|
|||
def __init__(self, llm: str = "qwen3_30b_instruct", **kwargs):
|
||||
# Configure for balanced performance
|
||||
simple_config = {
|
||||
"success_rate": 0.8, # Very high success rate
|
||||
"extra_time": 0.8, # Moderate speed (1.0-1.5s range)
|
||||
"relevance_ratio": 0.8, # High relevance
|
||||
"success_rate": 0.3, # Very high success rate
|
||||
"extra_time": 0, # Moderate speed (1.0-1.5s range)
|
||||
"relevance_ratio": 0.3, # High relevance
|
||||
"content_length": "medium" # More detailed than needed for simple
|
||||
}
|
||||
|
||||
medium_config = {
|
||||
"success_rate": 0.8, # Excellent success rate
|
||||
"extra_time": 1.0, # Balanced speed
|
||||
"relevance_ratio": 0.8, # High relevance
|
||||
"success_rate": 0.9, # Excellent success rate
|
||||
"extra_time": 0, # Balanced speed
|
||||
"relevance_ratio": 0.9, # High relevance
|
||||
"content_length": "medium" # Perfect depth for medium queries
|
||||
}
|
||||
|
||||
complex_config = {
|
||||
"success_rate": 0.8, # Good success rate
|
||||
"extra_time": 1.2, # Still reasonable speed
|
||||
"relevance_ratio": 0.8, # Decent relevance but not exhaustive
|
||||
"success_rate": 0.5, # Good success rate
|
||||
"extra_time": 0, # Still reasonable speed
|
||||
"relevance_ratio": 0.5, # Decent relevance but not exhaustive
|
||||
"content_length": "medium" # Covers main points but lacks depth
|
||||
}
|
||||
|
||||
|
|
@ -83,23 +83,23 @@ class SearchToolC(LLMMockSearchOp):
|
|||
def __init__(self, llm: str = "qwen3_30b_instruct", **kwargs):
|
||||
# Configure for comprehensive but costly performance
|
||||
simple_config = {
|
||||
"success_rate": 0.7, # Good but not optimal (over-processing)
|
||||
"extra_time": 2.5, # Slow (3.0-4.0s range)
|
||||
"relevance_ratio": 0.7, # High relevance but unnecessary depth
|
||||
"success_rate": 0.3, # Good but not optimal (over-processing)
|
||||
"extra_time": 0, # Slow (3.0-4.0s range)
|
||||
"relevance_ratio": 0.3, # High relevance but unnecessary depth
|
||||
"content_length": "long" # Too detailed for simple queries
|
||||
}
|
||||
|
||||
medium_config = {
|
||||
"success_rate": 0.7, # High success rate
|
||||
"extra_time": 2.8, # Slow but thorough
|
||||
"relevance_ratio": 0.7, # High relevance with extra context
|
||||
"success_rate": 0.4, # High success rate
|
||||
"extra_time": 0, # Slow but thorough
|
||||
"relevance_ratio": 0.4, # High relevance with extra context
|
||||
"content_length": "long" # More depth than needed
|
||||
}
|
||||
|
||||
complex_config = {
|
||||
"success_rate": 0.95, # Excellent success rate
|
||||
"extra_time": 3.5, # Slow but comprehensive (3.5-5.0s range)
|
||||
"relevance_ratio": 0.94, # Very high relevance
|
||||
"success_rate": 0.9, # Excellent success rate
|
||||
"extra_time": 0, # Slow but comprehensive (3.5-5.0s range)
|
||||
"relevance_ratio": 0.9, # Very high relevance
|
||||
"content_length": "long" # Perfect depth for complex queries
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,132 +0,0 @@
|
|||
"""
|
||||
Test script for UseMockSearchOp
|
||||
|
||||
This script demonstrates how to use the UseMockSearchOp to intelligently
|
||||
select and execute search tools based on query complexity.
|
||||
"""
|
||||
import asyncio
|
||||
import json
|
||||
|
||||
from flowllm.context import FlowContext
|
||||
from flowllm.app import FlowLLMApp
|
||||
|
||||
from reme_ai.agent.tools.use_mock_search_op import UseMockSearchOp
|
||||
|
||||
|
||||
async def test_single_query(op: UseMockSearchOp, query: str):
|
||||
"""Test a single query and display results."""
|
||||
print(f"\n{'=' * 100}")
|
||||
print(f"Testing Query: {query}")
|
||||
print(f"{'=' * 100}")
|
||||
|
||||
context = FlowContext(query=query)
|
||||
await op.async_call(context=context)
|
||||
|
||||
result = json.loads(context.use_mock_search_result)
|
||||
|
||||
print(f"\n📊 Results:")
|
||||
print(f" Selected Tool: {result['selected_tool']}")
|
||||
print(f" Reasoning: {result['reasoning']}")
|
||||
print(f" Query Complexity: {result['complexity']}")
|
||||
print(f" Success: {result['success']}")
|
||||
print(f"\n📝 Content:")
|
||||
print(f" {result['content'][:500]}..." if len(result['content']) > 500 else f" {result['content']}")
|
||||
print(f"\n{'=' * 100}\n")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
async def main():
|
||||
"""Run comprehensive tests of UseMockSearchOp."""
|
||||
|
||||
print("\n" + "=" * 100)
|
||||
print("UseMockSearchOp Test Suite")
|
||||
print("=" * 100)
|
||||
print("\nThis test demonstrates the intelligent tool selection capability.")
|
||||
print("The LLM analyzes each query and selects the most appropriate search tool:\n")
|
||||
print(" • SearchToolA: Fast & shallow (best for simple factual queries)")
|
||||
print(" • SearchToolB: Balanced (best for medium complexity queries)")
|
||||
print(" • SearchToolC: Comprehensive & slow (best for complex research queries)")
|
||||
print("=" * 100)
|
||||
|
||||
async with FlowLLMApp(load_default_config=True):
|
||||
op = UseMockSearchOp()
|
||||
|
||||
# Test cases covering different complexities
|
||||
test_queries = [
|
||||
# Simple queries (should select SearchToolA)
|
||||
{
|
||||
"query": "What is the capital of France?",
|
||||
"expected_tool": "SearchToolA",
|
||||
"description": "Simple factual query"
|
||||
},
|
||||
{
|
||||
"query": "When was Python programming language created?",
|
||||
"expected_tool": "SearchToolA",
|
||||
"description": "Simple historical fact"
|
||||
},
|
||||
|
||||
# Medium complexity queries (should select SearchToolB)
|
||||
{
|
||||
"query": "How does quantum computing work?",
|
||||
"expected_tool": "SearchToolB",
|
||||
"description": "Medium complexity technical explanation"
|
||||
},
|
||||
{
|
||||
"query": "What are the main causes of climate change?",
|
||||
"expected_tool": "SearchToolB",
|
||||
"description": "Medium complexity scientific question"
|
||||
},
|
||||
|
||||
# Complex queries (should select SearchToolC)
|
||||
{
|
||||
"query": "Analyze the impact of artificial intelligence on global economy, employment, and society",
|
||||
"expected_tool": "SearchToolC",
|
||||
"description": "Complex multi-dimensional analysis"
|
||||
},
|
||||
{
|
||||
"query": "Compare and contrast different renewable energy solutions including their environmental impact, cost-effectiveness, and scalability",
|
||||
"expected_tool": "SearchToolC",
|
||||
"description": "Complex comparative analysis"
|
||||
}
|
||||
]
|
||||
|
||||
results = []
|
||||
correct_selections = 0
|
||||
|
||||
for test_case in test_queries:
|
||||
query = test_case["query"]
|
||||
expected = test_case["expected_tool"]
|
||||
description = test_case["description"]
|
||||
|
||||
print(f"\n🔍 Test Case: {description}")
|
||||
print(f" Expected Tool: {expected}")
|
||||
|
||||
result = await test_single_query(op, query)
|
||||
results.append({
|
||||
"test_case": test_case,
|
||||
"result": result
|
||||
})
|
||||
|
||||
# Check if the selection was correct
|
||||
if result["selected_tool"] == expected:
|
||||
correct_selections += 1
|
||||
print(f" ✅ Correct tool selected!")
|
||||
else:
|
||||
print(f" ⚠️ Different tool selected (this may still be valid)")
|
||||
|
||||
# Summary
|
||||
print("\n" + "=" * 100)
|
||||
print("Test Summary")
|
||||
print("=" * 100)
|
||||
print(f"Total Tests: {len(test_queries)}")
|
||||
print(f"Expected Matches: {correct_selections}/{len(test_queries)}")
|
||||
print(f"Success Rate: {correct_selections/len(test_queries)*100:.1f}%")
|
||||
print("\nNote: Tool selection may vary based on LLM reasoning, and different")
|
||||
print(" selections don't necessarily indicate errors.")
|
||||
print("=" * 100)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
|
||||
|
|
@ -1,17 +1,18 @@
|
|||
import asyncio
|
||||
import datetime
|
||||
import json
|
||||
from typing import Dict, Any
|
||||
|
||||
from flowllm.op.gallery.task_react_op import tools_schema_to_qwen_prompt
|
||||
from flowllm.schema.tool_call import ToolCall
|
||||
from loguru import logger
|
||||
|
||||
from flowllm.context import FlowContext, C
|
||||
from flowllm.context import C
|
||||
from flowllm.enumeration.role import Role
|
||||
from flowllm.op.base_async_tool_op import BaseAsyncToolOp
|
||||
from flowllm.schema.message import Message
|
||||
from flowllm.schema.tool_call import ToolCall
|
||||
from flowllm.utils.timer import Timer
|
||||
from flowllm.utils.token_utils import TokenCounter
|
||||
from loguru import logger
|
||||
|
||||
from reme_ai.agent.tools.mock_search_tools import SearchToolA, SearchToolB, SearchToolC
|
||||
from reme_ai.schema.memory import ToolCallResult
|
||||
|
||||
|
||||
@C.register_op()
|
||||
|
|
@ -19,18 +20,30 @@ class UseMockSearchOp(BaseAsyncToolOp):
|
|||
file_path: str = __file__
|
||||
|
||||
def __init__(self, llm: str = "qwen3_30b_instruct", **kwargs):
|
||||
super().__init__(llm=llm, **kwargs)
|
||||
super().__init__(llm=llm, save_answer=True, **kwargs)
|
||||
|
||||
def build_tool_call(self) -> ToolCall:
|
||||
return ToolCall(**{
|
||||
"description": "Intelligently selects and executes the most appropriate search tool based on query complexity. "
|
||||
"Automatically tracks performance metrics and records tool usage for optimization.",
|
||||
"input_schema": {
|
||||
"query": {
|
||||
"type": "string",
|
||||
"description": "query",
|
||||
"required": True
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
async def select_tool(self, query: str, tool_ops: list[BaseAsyncToolOp]) -> ToolCall | None:
|
||||
assistant_message = await self.llm.achat(messages=[Message(role=Role.USER, content=query)],
|
||||
tools=[x.tool_call for x in tool_ops])
|
||||
|
||||
logger.info(f"assistant_message={assistant_message.model_dump_json()}")
|
||||
if assistant_message.tool_calls:
|
||||
return assistant_message.tool_calls[0]
|
||||
|
||||
return None
|
||||
|
||||
|
||||
async def async_execute(self):
|
||||
query: str = self.input_dict["query"]
|
||||
logger.info(f"query={query}")
|
||||
|
|
@ -41,46 +54,85 @@ class UseMockSearchOp(BaseAsyncToolOp):
|
|||
SearchToolC(),
|
||||
]
|
||||
|
||||
tool_result = await self.select_tool(query, tool_ops)
|
||||
if tool_result is None:
|
||||
...
|
||||
# Step 1: Select the appropriate tool using LLM
|
||||
tool_call = await self.select_tool(query, tool_ops)
|
||||
|
||||
if tool_call is None:
|
||||
# No tool selected
|
||||
error_result = ToolCallResult(
|
||||
create_time=datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
|
||||
tool_name="None",
|
||||
input={"query": query},
|
||||
output="No appropriate tool was selected for the query",
|
||||
token_cost=0,
|
||||
success=False,
|
||||
time_cost=0.0
|
||||
)
|
||||
self.set_result(error_result.model_dump_json())
|
||||
return
|
||||
|
||||
# Step 2: Execute the selected tool
|
||||
selected_op = None
|
||||
for op in tool_ops:
|
||||
op.tool_call.name == tool_result.name
|
||||
if op.tool_call.name == tool_call.name:
|
||||
selected_op = op
|
||||
break
|
||||
|
||||
if selected_op is None:
|
||||
# Tool not found (should not happen)
|
||||
error_result = ToolCallResult(
|
||||
create_time=datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
|
||||
tool_name=tool_call.name,
|
||||
input=tool_call.arguments,
|
||||
output=f"Tool {tool_call.name} not found in available tools",
|
||||
token_cost=0,
|
||||
success=False,
|
||||
time_cost=0.0
|
||||
)
|
||||
self.set_result(error_result.model_dump_json())
|
||||
return
|
||||
|
||||
# Step 3: Execute the tool with timer
|
||||
timer = Timer("tool execute")
|
||||
with timer:
|
||||
await selected_op.async_call(query=query)
|
||||
selected_op_output = json.loads(selected_op.output)
|
||||
content = selected_op_output["content"]
|
||||
success = selected_op_output["success"]
|
||||
token_cost = TokenCounter().count(content)
|
||||
|
||||
time_cost = timer.time_cost
|
||||
|
||||
# Create ToolCallResult
|
||||
tool_call_result = ToolCallResult(
|
||||
create_time=datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
|
||||
tool_name=tool_call.name,
|
||||
input={"query": query},
|
||||
output=content,
|
||||
token_cost=token_cost,
|
||||
success=success,
|
||||
time_cost=round(time_cost, 3)
|
||||
)
|
||||
|
||||
self.set_result(tool_call_result.model_dump_json())
|
||||
|
||||
|
||||
async def async_main():
|
||||
"""Test the UseMockSearchOp with different query types."""
|
||||
from flowllm.app import FlowLLMApp
|
||||
|
||||
async with FlowLLMApp(load_default_config=True):
|
||||
# Test queries of different complexities
|
||||
test_queries = [
|
||||
"What is the capital of France?", # Simple - should select SearchToolA
|
||||
"How does quantum computing work?", # Medium - should select SearchToolB
|
||||
"Analyze the impact of artificial intelligence on global economy, employment, and society", # Complex - should select SearchToolC
|
||||
"When was Python programming language created?", # Simple
|
||||
"Compare different types of renewable energy sources", # Complex
|
||||
"What is the capital of France?",
|
||||
"How does quantum computing work?",
|
||||
"Analyze the impact of artificial intelligence on global economy, employment, and society",
|
||||
"When was Python programming language created?",
|
||||
"Compare different types of renewable energy sources",
|
||||
]
|
||||
|
||||
op = UseMockSearchOp()
|
||||
|
||||
for query in test_queries:
|
||||
print(f"\n{'=' * 100}")
|
||||
print(f"Query: {query}")
|
||||
print(f"{'=' * 100}")
|
||||
|
||||
context = FlowContext(query=query)
|
||||
await op.async_call(context=context)
|
||||
|
||||
result = json.loads(context.use_mock_search_result)
|
||||
print(f"\nSelected Tool: {result['selected_tool']}")
|
||||
print(f"Reasoning: {result['reasoning']}")
|
||||
print(f"Complexity: {result['complexity']}")
|
||||
print(f"Success: {result['success']}")
|
||||
print(f"\nContent:\n{result['content']}")
|
||||
print(f"\n{'=' * 100}\n")
|
||||
op = UseMockSearchOp()
|
||||
await op.async_call(query=query)
|
||||
print(op.output)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
|
|
@ -1,64 +0,0 @@
|
|||
tool_selection_prompt: |
|
||||
You are an intelligent tool selection assistant. You have access to three different search tools with different characteristics:
|
||||
|
||||
**SearchToolA** - Fast but shallow search tool
|
||||
- Best for: Simple factual queries
|
||||
- Strengths: Very fast response (0.2-0.5s), high success rate for simple queries (95%)
|
||||
- Weaknesses: Poor performance on complex queries (50% success rate), limited depth
|
||||
|
||||
**SearchToolB** - Balanced search tool
|
||||
- Best for: Medium complexity queries
|
||||
- Strengths: Good balance of speed and quality (1.0-1.5s), consistent 80% success rate across all query types
|
||||
- Weaknesses: May be slower than needed for simple queries, lacks depth for very complex topics
|
||||
|
||||
**SearchToolC** - Comprehensive but slow search tool
|
||||
- Best for: Complex research queries
|
||||
- Strengths: Excellent for complex queries (95% success rate), comprehensive and detailed results
|
||||
- Weaknesses: Very slow (3.0-5.0s), overkill for simple queries (70% success rate)
|
||||
|
||||
Given the user's query, you need to:
|
||||
1. Analyze the complexity and requirements of the query
|
||||
2. Select the most appropriate tool (SearchToolA, SearchToolB, or SearchToolC)
|
||||
3. Provide the query parameter
|
||||
|
||||
User Query: {query}
|
||||
|
||||
Respond with a JSON object in the following format:
|
||||
{{
|
||||
"selected_tool": "SearchToolA" or "SearchToolB" or "SearchToolC",
|
||||
"reasoning": "Brief explanation of why this tool was selected",
|
||||
"query": "The search query to use"
|
||||
}}
|
||||
|
||||
tool_selection_prompt_zh: |
|
||||
你是一个智能工具选择助手。你可以访问三个具有不同特性的搜索工具：
|
||||
|
||||
**SearchToolA** - 快速但浅层的搜索工具
|
||||
- 最适合：简单的事实查询
|
||||
- 优势：响应非常快（0.2-0.5秒），简单查询的成功率高（95%）
|
||||
- 劣势：复杂查询表现不佳（成功率50%），深度有限
|
||||
|
||||
**SearchToolB** - 平衡的搜索工具
|
||||
- 最适合：中等复杂度的查询
|
||||
- 优势：速度和质量平衡良好（1.0-1.5秒），所有查询类型的成功率一致为80%
|
||||
- 劣势：对于简单查询可能比需要的慢，对于非常复杂的主题缺乏深度
|
||||
|
||||
**SearchToolC** - 全面但慢速的搜索工具
|
||||
- 最适合：复杂的研究查询
|
||||
- 优势：对复杂查询效果极佳（成功率95%），结果全面详细
|
||||
- 劣势：非常慢（3.0-5.0秒），对简单查询来说过度（成功率70%）
|
||||
|
||||
给定用户的查询，你需要：
|
||||
1. 分析查询的复杂度和要求
|
||||
2. 选择最合适的工具（SearchToolA、SearchToolB 或 SearchToolC）
|
||||
3. 提供查询参数
|
||||
|
||||
用户查询：{query}
|
||||
|
||||
用 JSON 对象格式回复：
|
||||
{{
|
||||
"selected_tool": "SearchToolA" 或 "SearchToolB" 或 "SearchToolC",
|
||||
"reasoning": "选择此工具的简要说明",
|
||||
"query": "要使用的搜索查询"
|
||||
}}
|
||||
|
||||
|
|
@ -1,8 +1,4 @@
|
|||
import sys
|
||||
import warnings
|
||||
|
||||
warnings.filterwarnings("ignore", category=DeprecationWarning, module="websockets")
|
||||
warnings.filterwarnings("ignore", category=DeprecationWarning, module="uvicorn")
|
||||
|
||||
from flowllm.app import FlowLLMApp
|
||||
|
||||
|
|
|
|||
|
|
@ -62,10 +62,6 @@ flow:
|
|||
flow_content: parse_tool_call_result_op >> update_vector_store_op
|
||||
description: "Evaluates and adds tool call results to the tool memory database, creating new memory or updating existing memory for the specified tool"
|
||||
input_schema:
|
||||
tool_name:
|
||||
type: string
|
||||
description: "The name of the tool to add call results for"
|
||||
required: true
|
||||
tool_call_results:
|
||||
type: array
|
||||
description: "List of tool call result objects, each containing: tool_name, input, output, success, time_cost, token_cost, create_time"
|
||||
|
|
@ -80,6 +76,15 @@ flow:
|
|||
description: "Comma-separated tool names to summarize (e.g., 'tool_name1,tool_name2')"
|
||||
required: true
|
||||
|
||||
use_mock_search:
|
||||
flow_content: use_mock_search_op
|
||||
description: "Simulates intelligent search tool selection and execution based on query complexity, with automatic tool memory recording"
|
||||
input_schema:
|
||||
query:
|
||||
type: string
|
||||
description: "User search query to process"
|
||||
required: true
|
||||
|
||||
summary_task_memory_simple:
|
||||
flow_content: simple_summary_op >> update_vector_store_op
|
||||
description: "Summarizes conversation trajectories or messages into memories using a simplified approach"
|
||||
|
|
|
|||
|
|
@ -0,0 +1 @@
|
|||
from .retrieve_tool_memory_op import RetrieveToolMemoryOp
|
||||
|
|
@ -117,7 +117,7 @@ class ToolCallResult(BaseModel):
|
|||
time_cost: float = Field(default=0, description="Time consumed by the tool invocation, in seconds")
|
||||
summary: str = Field(default="", description="Brief summary of the tool call result")
|
||||
evaluation: str = Field(default="", description="Detailed evaluation for the tool invocation")
|
||||
score: float = Field(default=0, description="Score of the Evaluation (0.0, 0.5, or 1.0)")
|
||||
score: float = Field(default=0, description="Score of the Evaluation (0.0 for failure, 1.0 for complete success)")
|
||||
|
||||
metadata: dict = Field(default_factory=dict)
|
||||
|
||||
|
|
@ -193,11 +193,9 @@ class ToolMemory(BaseMemory):
|
|||
avg_score = total_score / recent_calls_count if recent_calls_count > 0 else 0.0
|
||||
|
||||
return {
|
||||
"total_calls": total_calls,
|
||||
"recent_calls_analyzed": recent_calls_count,
|
||||
"avg_token_cost": round(avg_token_cost, 2),
|
||||
"success_rate": round(success_rate, 4),
|
||||
"avg_time_cost": round(avg_time_cost, 3),
|
||||
"success_rate": round(success_rate, 4),
|
||||
"avg_score": round(avg_score, 3)
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
import asyncio
|
||||
from collections import defaultdict
|
||||
from typing import List
|
||||
|
||||
from flowllm import C, BaseAsyncOp
|
||||
|
|
@ -44,18 +45,19 @@ class ParseToolCallResultOp(BaseAsyncOp):
|
|||
tool_call_result.evaluation = eval_data.get("evaluation", "")
|
||||
tool_call_result.score = float(eval_data.get("score", 0.0))
|
||||
|
||||
# 验证 score 是否符合 3 档要求 (0.0, 0.5, 1.0)
|
||||
if tool_call_result.score not in [0.0, 0.5, 1.0]:
|
||||
logger.warning(f"Score {tool_call_result.score} not in [0.0, 0.5, 1.0], rounding to nearest")
|
||||
if tool_call_result.score < 0.25:
|
||||
# 验证 score 是否符合 2 档要求 (0.0, 1.0)
|
||||
if tool_call_result.score not in [0.0, 1.0]:
|
||||
if tool_call_result.score < 0.5:
|
||||
tool_call_result.score = 0.0
|
||||
elif tool_call_result.score < 0.75:
|
||||
tool_call_result.score = 0.5
|
||||
else:
|
||||
tool_call_result.score = 1.0
|
||||
|
||||
logger.info(f"Evaluated tool call {index}: tool_name={tool_call_result.tool_name}, "
|
||||
f"score={tool_call_result.score}, summary={tool_call_result.summary[:50]}...")
|
||||
# 打印完整的prompt和result
|
||||
logger.info(f"\n{'='*80}\nLLM Evaluation [Index {index}]\n{'='*80}\n"
|
||||
f"PROMPT:\n{prompt}\n\n"
|
||||
f"RESULT:\n{content}\n"
|
||||
f"{'='*80}\n")
|
||||
|
||||
return tool_call_result
|
||||
|
||||
# 调用 LLM 进行评估
|
||||
|
|
@ -66,79 +68,68 @@ class ParseToolCallResultOp(BaseAsyncOp):
|
|||
async def async_execute(self):
|
||||
tool_call_results: list = self.context.get("tool_call_results", [])
|
||||
tool_call_results = [ToolCallResult(**x) if isinstance(x, dict) else x for x in tool_call_results]
|
||||
tool_name: str = self.context.get("tool_name", "")
|
||||
workspace_id: str = self.context.workspace_id
|
||||
logger.info(f"workspace_id={workspace_id} count={len(tool_call_results)} tool_name={tool_name}")
|
||||
|
||||
if not tool_name:
|
||||
logger.warning("tool_name is empty, skipping processing")
|
||||
self.context.response.answer = "tool_name is required"
|
||||
self.context.response.success = False
|
||||
return
|
||||
|
||||
if not tool_call_results:
|
||||
logger.info("No valid tool_call_results to process")
|
||||
self.context.response.answer = "No valid tool_call_results"
|
||||
self.context.response.success = False
|
||||
return
|
||||
|
||||
# 并发评估所有 tool_call_results
|
||||
logger.info(f"Starting concurrent evaluation of {len(tool_call_results)} tool call results")
|
||||
|
||||
# 使用基类的 submit_async_task 提交所有评估任务
|
||||
for index, tool_call_result in enumerate(tool_call_results):
|
||||
self.submit_async_task(self._evaluate_single_tool_call, tool_call_result, index)
|
||||
|
||||
# 使用基类的 join_async_task 等待所有任务完成
|
||||
# 注意: 基类已经过滤掉异常,返回的只包含成功的结果
|
||||
tool_call_results = await self.join_async_task(return_exceptions=True)
|
||||
logger.info(f"Completed evaluation of {len(tool_call_results)} tool call results")
|
||||
evaluated_results = await self.join_async_task(return_exceptions=True)
|
||||
|
||||
nodes: List[VectorNode] = await self.vector_store.async_search(query=tool_name,
|
||||
workspace_id=workspace_id,
|
||||
top_k=1)
|
||||
tool_results_by_name = defaultdict(list)
|
||||
for result in evaluated_results:
|
||||
tool_results_by_name[result.tool_name].append(result)
|
||||
|
||||
tool_memory: ToolMemory | None = None
|
||||
exist_node: bool = False
|
||||
# 处理每个 tool_name 的结果
|
||||
all_memory_list = []
|
||||
all_deleted_memory_ids = []
|
||||
|
||||
if nodes:
|
||||
top_node = nodes[0]
|
||||
memory: ToolMemory = vector_node_to_memory(top_node)
|
||||
for tool_name, tool_call_results in tool_results_by_name.items():
|
||||
nodes: List[VectorNode] = await self.vector_store.async_search(query=tool_name,
|
||||
workspace_id=workspace_id,
|
||||
top_k=1)
|
||||
|
||||
# 确保是 ToolMemory 类型且 when_to_use 与 tool_name 匹配
|
||||
if isinstance(memory, ToolMemory) and memory.when_to_use == tool_name:
|
||||
tool_memory = memory
|
||||
exist_node = True
|
||||
logger.info(f"Found existing tool_memory for tool_name={tool_name}, memory_id={tool_memory.memory_id}")
|
||||
else:
|
||||
logger.info(f"Top result does not match tool_name={tool_name}, will create new memory")
|
||||
tool_memory: ToolMemory | None = None
|
||||
exist_node: bool = False
|
||||
|
||||
# 如果没有找到匹配的 memory，创建新的
|
||||
if tool_memory is None:
|
||||
tool_memory = ToolMemory(workspace_id=workspace_id, when_to_use=tool_name)
|
||||
logger.info(f"Created new tool_memory for tool_name={tool_name}, memory_id={tool_memory.memory_id}")
|
||||
if nodes:
|
||||
top_node = nodes[0]
|
||||
memory: ToolMemory = vector_node_to_memory(top_node)
|
||||
|
||||
tool_memory.tool_call_results.extend(tool_call_results)
|
||||
# 确保是 ToolMemory 类型且 when_to_use 与 tool_name 匹配
|
||||
if isinstance(memory, ToolMemory) and memory.when_to_use == tool_name:
|
||||
tool_memory = memory
|
||||
exist_node = True
|
||||
|
||||
# 保留最近的 n 个
|
||||
if len(tool_memory.tool_call_results) > self.max_history_tool_call_cnt:
|
||||
tool_memory.tool_call_results = tool_memory.tool_call_results[-self.max_history_tool_call_cnt:]
|
||||
logger.info(f"Trimmed tool_call_results to {self.max_history_tool_call_cnt} most recent entries")
|
||||
# 如果没有找到匹配的 memory，创建新的
|
||||
if tool_memory is None:
|
||||
tool_memory = ToolMemory(workspace_id=workspace_id, when_to_use=tool_name)
|
||||
|
||||
# 更新修改时间
|
||||
tool_memory.update_modified_time()
|
||||
tool_memory.tool_call_results.extend(tool_call_results)
|
||||
|
||||
# 4. 将更新后的 memory 保存到向量数据库
|
||||
# 如果是更新现有的 memory，需要先删除旧的
|
||||
deleted_memory_ids = []
|
||||
if exist_node:
|
||||
deleted_memory_ids = [tool_memory.memory_id]
|
||||
# 保留最近的 n 个
|
||||
if len(tool_memory.tool_call_results) > self.max_history_tool_call_cnt:
|
||||
tool_memory.tool_call_results = tool_memory.tool_call_results[-self.max_history_tool_call_cnt:]
|
||||
|
||||
# 更新修改时间
|
||||
tool_memory.update_modified_time()
|
||||
|
||||
# 如果是更新现有的 memory，需要先删除旧的
|
||||
if exist_node:
|
||||
all_deleted_memory_ids.append(tool_memory.memory_id)
|
||||
|
||||
all_memory_list.append(tool_memory)
|
||||
|
||||
# 设置返回结果
|
||||
self.context.response.metadata["deleted_memory_ids"] = deleted_memory_ids
|
||||
self.context.response.metadata["memory_list"] = [tool_memory]
|
||||
|
||||
logger.info(f"Updated tool_memory: tool_name={tool_name}, total_results={len(tool_memory.tool_call_results)}")
|
||||
self.context.response.metadata["deleted_memory_ids"] = all_deleted_memory_ids
|
||||
self.context.response.metadata["memory_list"] = all_memory_list
|
||||
|
||||
|
||||
async def main():
|
||||
|
|
@ -166,10 +157,9 @@ async def main():
|
|||
"time_cost": 2.3
|
||||
}
|
||||
]
|
||||
tool_name = "test_tool"
|
||||
workspace_id = "test_workspace1"
|
||||
|
||||
await op.async_call(tool_call_results=tool_call_results, tool_name=tool_name, workspace_id=workspace_id)
|
||||
await op.async_call(tool_call_results=tool_call_results, workspace_id=workspace_id)
|
||||
logger.info(f"Response: {op.context.response.model_dump_json()}")
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -14,17 +14,23 @@ evaluate_tool_call_prompt: |
|
|||
|
||||
## Evaluation Criteria:
|
||||
|
||||
### 1. Success Evaluation
|
||||
- Check if the success flag indicates successful execution
|
||||
- Verify if the output contains error messages or indicates failure
|
||||
- Consider if time and token costs are reasonable
|
||||
- Assess if there are any error indicators in the output
|
||||
### Important: Score independently from the success flag
|
||||
The `success_flag` indicates whether the tool executed without technical errors.
|
||||
The `score` should evaluate the QUALITY and RELEVANCE of the result.
|
||||
|
||||
### 2. Parameter Alignment Evaluation
|
||||
- Evaluate if the output matches what would be expected given the input parameters
|
||||
- Consider if the input parameters are appropriate for the tool
|
||||
- Check for any parameter mismatches or unexpected behaviors
|
||||
- Verify the output is consistent with the tool's intended purpose
|
||||
A tool can execute successfully (success=True) but still produce low-quality or irrelevant results (score=0.0).
|
||||
|
||||
### Evaluation Dimensions:
|
||||
|
||||
1. **Technical Execution** (reflected in success_flag):
|
||||
- Did the tool run without errors?
|
||||
- Are there error messages or exceptions in the output?
|
||||
|
||||
2. **Result Quality** (what you should score):
|
||||
- Is the output relevant to the input query/parameters?
|
||||
- Does the output provide meaningful and useful information?
|
||||
- Is the result appropriate given the input parameters?
|
||||
- Does the output match the tool's intended purpose?
|
||||
|
||||
## Response Format:
|
||||
Please provide your evaluation in the following JSON format:
|
||||
|
|
@ -32,15 +38,20 @@ evaluate_tool_call_prompt: |
|
|||
```json
|
||||
{{
|
||||
"summary": "A brief one-sentence summary of the tool call result",
|
||||
"evaluation": "A brief evaluation (2-3 sentences) explaining the success status and parameter alignment.",
|
||||
"evaluation": "A brief evaluation (2-3 sentences) explaining the result quality and relevance, NOT just repeating the success flag.",
|
||||
"score": 1.0
|
||||
}}
|
||||
```
|
||||
|
||||
## Scoring Guidelines:
|
||||
- **1.0**: Success - Tool executed successfully with good parameter alignment
|
||||
- **0.5**: Partial Success - Tool executed but with parameter alignment issues or minor errors
|
||||
- **0.0**: Failure - Tool execution failed or severe parameter misalignment
|
||||
## Scoring Guidelines (Focus on Result Quality):
|
||||
- **1.0**: High Quality - The output is relevant, useful, and appropriate for the given input parameters
|
||||
- **0.0**: Low Quality - The output is irrelevant, incorrect, unhelpful, or inappropriate for the input
|
||||
|
||||
Please evaluate carefully and provide a score of exactly 0.0, 0.5, or 1.0.
|
||||
## Examples:
|
||||
- success=True, but output is "No results found" for a reasonable query → score=0.0 (technically succeeded but unhelpful)
|
||||
- success=True, but output contains generic/irrelevant information → score=0.0 (poor quality)
|
||||
- success=True, and output provides relevant, useful information → score=1.0 (good quality)
|
||||
- success=False, with error messages → score=0.0 (failed execution)
|
||||
|
||||
Please evaluate carefully and provide a score of exactly 0.0 or 1.0 based on OUTPUT QUALITY, not just the success flag.
|
||||
|
||||
|
|
|
|||
|
|
@ -45,12 +45,8 @@ class SummaryToolMemoryOp(BaseAsyncOp):
|
|||
@staticmethod
|
||||
def _format_statistics_markdown(statistics: dict) -> str:
|
||||
"""Format statistics as markdown."""
|
||||
lines = [f"- **Total Calls**: {statistics.get('total_calls', 0)}",
|
||||
f"- **Recent Calls**: {statistics.get('recent_calls', 0)}",
|
||||
f"- **Success Rate**: {statistics.get('success_rate', 0):.2%}",
|
||||
f"- **Recent Success Rate**: {statistics.get('recent_success_rate', 0):.2%}",
|
||||
lines = [f"- **Success Rate**: {statistics.get('success_rate', 0):.2%}",
|
||||
f"- **Average Score**: {statistics.get('avg_score', 0):.3f}",
|
||||
f"- **Recent Average Score**: {statistics.get('recent_avg_score', 0):.3f}",
|
||||
f"- **Average Time Cost**: {statistics.get('avg_time_cost', 0):.3f}s",
|
||||
f"- **Average Token Cost**: {statistics.get('avg_token_cost', 0):.1f}"]
|
||||
|
||||
|
|
@ -73,15 +69,18 @@ class SummaryToolMemoryOp(BaseAsyncOp):
|
|||
call_summaries_md = self._format_call_summaries_markdown(recent_calls)
|
||||
statistics_md = self._format_statistics_markdown(statistics)
|
||||
|
||||
# Don't include statistics in prompt - only call summaries
|
||||
prompt = self.prompt_format(prompt_name="summarize_tool_usage_prompt",
|
||||
tool_name=tool_memory.when_to_use,
|
||||
call_summaries=call_summaries_md,
|
||||
statistics=statistics_md)
|
||||
call_summaries=call_summaries_md)
|
||||
|
||||
def parse_summary(message: Message) -> ToolMemory:
|
||||
content = message.content.strip()
|
||||
# Extract content from txt code block
|
||||
tool_memory.content = extract_content(content, "txt")
|
||||
llm_summary = extract_content(content, "txt")
|
||||
|
||||
# Append statistics markdown to LLM result
|
||||
tool_memory.content = f"{llm_summary}\n\n## Statistics\n{statistics_md}"
|
||||
|
||||
# Update modified time
|
||||
tool_memory.update_modified_time()
|
||||
|
|
@ -334,7 +333,7 @@ async def main():
|
|||
logger.info(f"\n工具名称: {summarized_memory.when_to_use}")
|
||||
logger.info(f"\n统计信息:")
|
||||
stats = summarized_memory.statistic(recent_frequency=30)
|
||||
logger.info(f" 总调用次数: {stats['total_calls']}")
|
||||
logger.info(f" 总调用次数: {len(summarized_memory.tool_call_results)}")
|
||||
logger.info(f" 成功率: {stats['success_rate']:.1%}")
|
||||
logger.info(f" 平均评分: {stats['avg_score']:.2f}")
|
||||
logger.info(f" 平均耗时: {stats['avg_time_cost']:.2f}s")
|
||||
|
|
|
|||
|
|
@ -7,14 +7,11 @@ summarize_tool_usage_prompt: |
|
|||
## Recent Tool Call History:
|
||||
{call_summaries}
|
||||
|
||||
## Statistical Summary:
|
||||
{statistics}
|
||||
|
||||
## Your Task:
|
||||
Based on the tool call history, generate a concise and logical tool usage description following this structure:
|
||||
|
||||
1. **Core Function**: What this tool does and when to use it
|
||||
2. **Success Patterns**: Parameter patterns and usage scenarios that work well (with key metrics)
|
||||
2. **Success Patterns**: Parameter patterns and usage scenarios that work well
|
||||
3. **Common Issues**: Main pitfalls to avoid and why they fail
|
||||
4. **Best Practices**: 2-3 actionable recommendations
|
||||
|
||||
|
|
|
|||
195
reme_ai/utils/tool_memory_utils.py
Normal file
195
reme_ai/utils/tool_memory_utils.py
Normal file
|
|
@ -0,0 +1,195 @@
|
|||
import random
|
||||
from datetime import datetime, timedelta
|
||||
from typing import List, Dict, Any
|
||||
|
||||
|
||||
def create_mock_tool_call_results(tool_name: str, count: int = 30) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Create mock tool call results for testing
|
||||
|
||||
Args:
|
||||
tool_name: Name of the tool to create results for
|
||||
count: Number of mock results to create
|
||||
|
||||
Returns:
|
||||
List of tool call result dictionaries
|
||||
"""
|
||||
base_time = datetime.now() - timedelta(days=7)
|
||||
tool_call_results = []
|
||||
|
||||
if tool_name == "web_search":
|
||||
# Scenario 1: Successful searches (15 calls)
|
||||
success_queries = [
|
||||
"Python asyncio tutorial", "machine learning basics", "React hooks guide",
|
||||
"Docker best practices", "SQL optimization tips", "Git workflow strategies",
|
||||
"RESTful API design", "microservices architecture", "Redis caching patterns",
|
||||
"Kubernetes deployment", "GraphQL advantages", "MongoDB schema design",
|
||||
"JWT authentication", "OAuth2 flow", "WebSocket real-time"
|
||||
]
|
||||
|
||||
for i, query in enumerate(success_queries):
|
||||
tool_call_results.append({
|
||||
"create_time": (base_time + timedelta(hours=i * 2)).strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"tool_name": tool_name,
|
||||
"input": {
|
||||
"query": query,
|
||||
"max_results": random.randint(5, 20),
|
||||
"language": "en",
|
||||
"filter_type": "technical_docs"
|
||||
},
|
||||
"output": f"Found {random.randint(8, 20)} relevant results for '{query}'. Top results include official documentation, tutorials, and best practice guides.",
|
||||
"token_cost": random.randint(100, 300),
|
||||
"success": True,
|
||||
"time_cost": round(random.uniform(1.5, 3.5), 2)
|
||||
})
|
||||
|
||||
# Scenario 2: Poor parameters (8 calls)
|
||||
for i in range(8):
|
||||
tool_call_results.append({
|
||||
"create_time": (base_time + timedelta(hours=30 + i * 3)).strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"tool_name": tool_name,
|
||||
"input": {
|
||||
"query": f"test query {i}",
|
||||
"max_results": 100,
|
||||
"language": "unknown",
|
||||
},
|
||||
"output": f"Warning: language 'unknown' not supported, using default. Query too generic, returning limited results. Found {random.randint(2, 5)} results.",
|
||||
"token_cost": random.randint(50, 150),
|
||||
"success": True,
|
||||
"time_cost": round(random.uniform(2.0, 4.0), 2)
|
||||
})
|
||||
|
||||
# Scenario 3: Timeouts or failures (5 calls)
|
||||
for i in range(5):
|
||||
tool_call_results.append({
|
||||
"create_time": (base_time + timedelta(hours=54 + i * 4)).strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"tool_name": tool_name,
|
||||
"input": {
|
||||
"query": f"extremely complex query with many conditions {i}",
|
||||
"max_results": 50,
|
||||
"language": "en",
|
||||
"filter_type": "all"
|
||||
},
|
||||
"output": "Error: Request timeout after 10 seconds. Try simplifying the query or reducing max_results.",
|
||||
"token_cost": 20,
|
||||
"success": False,
|
||||
"time_cost": 10.0
|
||||
})
|
||||
|
||||
# Scenario 4: Empty results (2 calls)
|
||||
for i in range(2):
|
||||
tool_call_results.append({
|
||||
"create_time": (base_time + timedelta(hours=74 + i * 5)).strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"tool_name": tool_name,
|
||||
"input": {
|
||||
"query": f"xyzabc123nonexistent{i}",
|
||||
"max_results": 10,
|
||||
"language": "en",
|
||||
},
|
||||
"output": "No results found for the given query. Please try different keywords.",
|
||||
"token_cost": 30,
|
||||
"success": True,
|
||||
"time_cost": 1.2
|
||||
})
|
||||
|
||||
elif tool_name == "database_query":
|
||||
# Scenario 1: Successful queries (12 calls)
|
||||
for i in range(12):
|
||||
tool_call_results.append({
|
||||
"create_time": (base_time + timedelta(hours=i * 3)).strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"tool_name": tool_name,
|
||||
"input": {
|
||||
"table": f"users_{i % 3}",
|
||||
"query": f"SELECT * FROM table WHERE id > {i * 10}",
|
||||
"limit": random.randint(10, 100)
|
||||
},
|
||||
"output": f"Query executed successfully. Returned {random.randint(5, 50)} rows in {round(random.uniform(0.1, 0.5), 3)}s.",
|
||||
"token_cost": random.randint(20, 80),
|
||||
"success": True,
|
||||
"time_cost": round(random.uniform(0.1, 0.5), 3)
|
||||
})
|
||||
|
||||
# Scenario 2: Slow queries (6 calls)
|
||||
for i in range(6):
|
||||
tool_call_results.append({
|
||||
"create_time": (base_time + timedelta(hours=36 + i * 4)).strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"tool_name": tool_name,
|
||||
"input": {
|
||||
"table": "large_table",
|
||||
"query": f"SELECT * FROM large_table WHERE name LIKE '%pattern%' ORDER BY created_at",
|
||||
"limit": 1000
|
||||
},
|
||||
"output": f"Query executed but took longer than expected. Returned {random.randint(100, 1000)} rows. Consider adding indexes.",
|
||||
"token_cost": random.randint(50, 150),
|
||||
"success": True,
|
||||
"time_cost": round(random.uniform(5.0, 10.0), 2)
|
||||
})
|
||||
|
||||
# Scenario 3: Query errors (4 calls)
|
||||
for i in range(4):
|
||||
tool_call_results.append({
|
||||
"create_time": (base_time + timedelta(hours=60 + i * 5)).strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"tool_name": tool_name,
|
||||
"input": {
|
||||
"table": "invalid_table",
|
||||
"query": f"SELECT * FROM invalid_table WHERE bad_column = {i}",
|
||||
"limit": 10
|
||||
},
|
||||
"output": f"Error: Table 'invalid_table' does not exist or column 'bad_column' not found.",
|
||||
"token_cost": 10,
|
||||
"success": False,
|
||||
"time_cost": 0.05
|
||||
})
|
||||
|
||||
elif tool_name == "file_processor":
|
||||
# Scenario 1: Successful file processing (10 calls)
|
||||
file_types = ["csv", "json", "xml", "txt", "pdf"]
|
||||
for i in range(10):
|
||||
file_type = file_types[i % len(file_types)]
|
||||
tool_call_results.append({
|
||||
"create_time": (base_time + timedelta(hours=i * 4)).strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"tool_name": tool_name,
|
||||
"input": {
|
||||
"file_path": f"/data/file_{i}.{file_type}",
|
||||
"operation": "read",
|
||||
"encoding": "utf-8"
|
||||
},
|
||||
"output": f"Successfully processed {file_type.upper()} file. Size: {random.randint(100, 5000)}KB, Records: {random.randint(100, 10000)}",
|
||||
"token_cost": random.randint(50, 200),
|
||||
"success": True,
|
||||
"time_cost": round(random.uniform(1.0, 3.0), 2)
|
||||
})
|
||||
|
||||
# Scenario 2: Large file warnings (5 calls)
|
||||
for i in range(5):
|
||||
tool_call_results.append({
|
||||
"create_time": (base_time + timedelta(hours=40 + i * 6)).strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"tool_name": tool_name,
|
||||
"input": {
|
||||
"file_path": f"/data/large_file_{i}.csv",
|
||||
"operation": "read",
|
||||
"encoding": "utf-8"
|
||||
},
|
||||
"output": f"Warning: Large file detected ({random.randint(50, 200)}MB). Processing may take longer. Consider using streaming mode.",
|
||||
"token_cost": random.randint(200, 500),
|
||||
"success": True,
|
||||
"time_cost": round(random.uniform(10.0, 30.0), 2)
|
||||
})
|
||||
|
||||
# Scenario 3: File not found (3 calls)
|
||||
for i in range(3):
|
||||
tool_call_results.append({
|
||||
"create_time": (base_time + timedelta(hours=70 + i * 8)).strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"tool_name": tool_name,
|
||||
"input": {
|
||||
"file_path": f"/invalid/path/file_{i}.txt",
|
||||
"operation": "read",
|
||||
"encoding": "utf-8"
|
||||
},
|
||||
"output": "Error: File not found. Please check the file path and ensure the file exists.",
|
||||
"token_cost": 10,
|
||||
"success": False,
|
||||
"time_cost": 0.01
|
||||
})
|
||||
|
||||
return tool_call_results[:count]
|
||||
Loading…
Add table
Reference in a new issue