diff --git a/README.md b/README.md index 1c909fc8..744242b1 100644 --- a/README.md +++ b/README.md @@ -59,6 +59,8 @@ memory, then continuously indexes, links, and consolidates that memory for futur ## 📰 News +- [2026.08] - Experience-driven enhancement of agent tool-use execution built on ReMe is available on + [arXiv:2608.03403](https://arxiv.org/abs/2608.03403). - [2026.07] - Introduced optional Cookbooks: [Daily Paper](cookbook/daily_paper/README.md) for paper discovery and analysis, and [Auto Fin](cookbook/auto-fin/README.md) for file-native ETF event research based on CLS news and historical market reactions. diff --git a/README_ZH.md b/README_ZH.md index 79ecf6f3..56c9898d 100644 --- a/README_ZH.md +++ b/README_ZH.md @@ -52,6 +52,8 @@ Agent 能够可靠召回。 ## 📰 新闻 +- [2026.08] - 基于 ReMe 的智能体工具使用执行增强工作见 + [arXiv:2608.03403](https://arxiv.org/abs/2608.03403)。 - [2026.07] - 新增可选 Cookbook 工作流：[每日论文](cookbook/daily_paper/README_ZH.md)用于论文发现与解析， [Auto Fin](cookbook/auto-fin/README_ZH.md)用于结合财联社新闻和历史行情开展文件化 ETF 事件研究。 - [2026.07] - diff --git a/benchmark/toolmemory/README.md b/benchmark/toolmemory/README.md new file mode 100644 index 00000000..7f358018 --- /dev/null +++ b/benchmark/toolmemory/README.md @@ -0,0 +1,98 @@ +## Towards Robust Tool Use in Agents via Experience-Driven Adaptive Guidance + +**Language**: English (default) / [中文](./README_ZH.md) + +> Paper: [arXiv:2608.03403](https://arxiv.org/abs/2608.03403) +> Code: [https://github.com/WangCan1178/ExpG](https://github.com/WangCan1178/ExpG) + +

+ ExpG challenges and overview +

+ +### Overview + +This folder archives **ExpG**, a tool-use enhancement built on [Agentscope ReMe](https://github.com/agentscope-ai/ReMe). ExpG mines, distills, and reuses experience from historical tool calls to provide **capability boundaries** and **best-practice guidance**, which helps agents: + +- Select and invoke tools more robustly under dynamic or noisy environments; +- Let smaller models with guidance outperform larger, memoryless baselines; +- Improve consistently across tool selection, tool calling, and response generation. + +**How ReMe is used:** Start the Tool Memory service; historical tool calls are written and evaluated via `add_tool_call_result`, distilled into tool-level guidance via `summary_tool_memory`, then retrieved and injected into later reasoning via `retrieve_tool_memory`. ReMe provides the vector store and service APIs; the acquisition / distillation / reuse strategy is implemented by ExpG. Full implementation and experiments are in [WangCan1178/ExpG](https://github.com/WangCan1178/ExpG). + +--- + +### ExpG Mechanism + +ExpG treats tool invocations as learnable experience and runs a three-stage pipeline: + +1. **Experience Acquisition** + - Analyze invocation quality from historical trajectories (success/failure, cost, latency, etc.); + - Build structured experience units per tool, recording context, parameter patterns, and outcomes. + +2. **Experience Distillation** + - Filter noisy or unhelpful experiences and keep representative patterns; + - Aggregate by equivalence classes to cover common and rare failure modes; + - Summarize with an LLM into generalizable textual guidance. + +3. **Experience Reuse** + - Retrieve relevant experience / guidance for future tasks; + - Inject guidance into tool selection, argument generation, and response synthesis; + - Improve stability under dynamic environments and imperfect feedback. + +--- + +### Main Results + +Performance comparison (%) across MetaTool, API-Bank, and BFCL-V3. **Bold** indicates the best results within each model. + +| Model | Method | MetaTool Pass@1 | MetaTool Avg@3 | MetaTool Pass@3 | API-Bank Pass@1 | API-Bank Avg@3 | API-Bank Pass@3 | BFCL-V3 Pass@1 | BFCL-V3 Avg@3 | BFCL-V3 Pass@3 | Total Pass@1 | Total Avg@3 | Total Pass@3 | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| GPT-5 nano | No Method | 72.62 | 72.76 | 78.49 | 82.96 | 83.46 | 86.97 | 53.80 | 53.00 | 60.95 | 70.82 | 70.62 | 76.63 | +| GPT-5 nano | Few-shot | 74.12 | 75.11 | 82.32 | 83.71 | 83.96 | **87.22** | 56.18 | 55.24 | 61.39 | 72.36 | 72.65 | 79.28 | +| GPT-5 nano | DRAFT | 73.94 | 73.04 | 78.97 | 84.21 | 83.46 | **87.22** | 57.27 | 57.27 | 62.26 | 72.52 | 71.58 | 77.23 | +| GPT-5 nano | Mem0 | 74.96 | 76.13 | 82.92 | 84.96 | 85.21 | **87.22** | 60.95 | 61.61 | 65.08 | 73.98 | 74.67 | 80.35 | +| GPT-5 nano | **ExpG** | **81.67** | **82.07** | **84.60** | **86.72** | **86.55** | **87.22** | **64.43** | **63.99** | **66.38** | **79.32** | **79.22** | **81.69** | +| DeepSeek-V3 | No Method | 83.10 | 82.94 | 84.66 | 84.71 | 84.38 | 85.46 | 58.79 | 59.65 | 65.94 | 78.92 | 78.66 | 81.37 | +| DeepSeek-V3 | Few-shot | 82.74 | 83.90 | 86.28 | 85.21 | 84.63 | 86.22 | 60.52 | 60.30 | 67.90 | 79.08 | 79.45 | 82.92 | +| DeepSeek-V3 | DRAFT | 80.23 | 80.79 | 82.44 | 84.96 | 85.63 | 86.47 | 62.26 | 61.61 | 68.55 | 77.70 | 77.80 | 80.54 | +| DeepSeek-V3 | Mem0 | 83.88 | 84.56 | 86.40 | 85.46 | 85.55 | 86.47 | 65.08 | 65.15 | 68.33 | 80.70 | 80.91 | 83.12 | +| DeepSeek-V3 | **ExpG** | **85.26** | **85.38** | **86.52** | **87.72** | **87.39** | **87.97** | **69.41** | **69.92** | **72.02** | **82.76** | **82.61** | **84.11** | +| Qwen3-8B | No Method | 76.51 | 76.97 | 77.71 | 83.96 | 83.88 | 84.21 | 58.79 | 58.28 | 60.30 | 74.46 | 74.41 | 75.56 | +| Qwen3-8B | Few-shot | 79.93 | 79.83 | 82.92 | 83.71 | 82.62 | 84.96 | 60.09 | 59.29 | 61.39 | 76.91 | 76.27 | 79.32 | +| Qwen3-8B | DRAFT | 78.19 | 77.33 | 77.89 | 85.71 | 84.96 | 85.46 | 60.74 | 60.30 | 62.91 | 76.20 | 75.18 | 76.35 | +| Qwen3-8B | Mem0 | 75.07 | 75.47 | 82.38 | 86.22 | 86.05 | 86.47 | 63.34 | 64.93 | 66.16 | 74.69 | 74.98 | 80.07 | +| Qwen3-8B | **ExpG** | **83.52** | **84.88** | **85.08** | **86.47** | **87.89** | **87.97** | **67.46** | **66.96** | **68.33** | **81.06** | **81.82** | **82.48** | +| Qwen3-32B | No Method | 80.05 | 79.43 | 80.17 | 84.71 | 84.88 | 85.21 | 65.15 | 65.08 | 66.16 | 78.05 | 77.55 | 78.41 | +| Qwen3-32B | **ExpG** | **84.68** | **85.02** | **86.28** | **86.97** | **87.30** | **87.72** | **70.72** | **71.01** | **73.32** | **82.48** | **82.56** | **84.14** | +| Qwen3-235B | No Method | 78.25 | 79.23 | 80.29 | 85.46 | 85.46 | 85.71 | 71.37 | 71.15 | 73.54 | 78.13 | 78.49 | 79.91 | +| Qwen3-235B | **ExpG** | **86.34** | **86.70** | **86.94** | **87.47** | **86.97** | **88.22** | **79.61** | **78.52** | **80.04** | **85.29** | **84.98** | **85.69** | + +--- + +### Reference Code + +| Path | Role | +| --- | --- | +| [`tool_memory.py`](./tool_memory.py) | HTTP client for official ReMe Tool Memory APIs (`add_tool_call_result` / `summary_tool_memory` / `retrieve_tool_memory`) | +| [`parse_tool_call_result_prompt.yaml`](./parse_tool_call_result_prompt.yaml) | Prompt for multi-aspect evaluation of each tool call | +| [`summary_tool_memory_prompt.yaml`](./summary_tool_memory_prompt.yaml) | Prompt for summarizing tool call history into guidance | +| [`tool_memory_flows.yaml`](./tool_memory_flows.yaml) | Tool Memory flow / op config excerpt | + +These are reference snippets. For the full runnable codebase, see [WangCan1178/ExpG](https://github.com/WangCan1178/ExpG). + +--- + +### Citation + +```bibtex +@misc{wang2026expg, + title = {Towards Robust Tool Use in Agents via Experience-Driven Adaptive Guidance}, + author = {Can Wang and Haoran Chen and Li Yu and Ding Hao and Bohai Zhao and Zhaoyang Liu and Zhiying Tu}, + year = {2026}, + eprint = {2608.03403}, + archivePrefix = {arXiv}, + primaryClass = {cs.AI}, + url = {https://arxiv.org/abs/2608.03403}, + howpublished = {\url{https://github.com/WangCan1178/ExpG}} +} +``` diff --git a/benchmark/toolmemory/README_ZH.md b/benchmark/toolmemory/README_ZH.md new file mode 100644 index 00000000..b0c5b89b --- /dev/null +++ b/benchmark/toolmemory/README_ZH.md @@ -0,0 +1,98 @@ +## Towards Robust Tool Use in Agents via Experience-Driven Adaptive Guidance + +**语言**：中文 / [English](./README.md) + +> 论文：[arXiv:2608.03403](https://arxiv.org/abs/2608.03403) +> 代码：[https://github.com/WangCan1178/ExpG](https://github.com/WangCan1178/ExpG) + +

+ ExpG 挑战与概览 +

+ +### 简介 + +本目录归档基于 [Agentscope ReMe](https://github.com/agentscope-ai/ReMe) 的工具使用增强工作 **ExpG**：在 ReMe 记忆框架之上，从历史工具调用中挖掘、提炼并复用经验，为智能体提供工具的 **能力边界** 与 **最佳实践指导**，从而： + +- 在动态或有噪环境下更鲁棒地选择和调用工具； +- 让较小模型在带有经验指导时超越更大、但无记忆的基线； +- 在工具选择、工具调用和响应生成等多个阶段带来一致收益。 + +**如何使用 ReMe：** 启动 Tool Memory 服务后，历史工具调用经 `add_tool_call_result` 写入并评估，经 `summary_tool_memory` 蒸馏成工具级指导，再经 `retrieve_tool_memory` 取回并注入后续推理。向量存储与服务接口由 ReMe 提供，经验获取 / 蒸馏 / 复用策略由 ExpG 实现。完整实现与实验见 [WangCan1178/ExpG](https://github.com/WangCan1178/ExpG)。 + +--- + +### ExpG 机制概览 + +ExpG 将工具调用视为可学习经验，并通过三阶段流水线完成经验的获取、提炼与复用： + +1. **经验获取（Experience Acquisition）** + - 从历史工具调用轨迹中分析调用质量（成功/失败、代价、时间等）； + - 针对不同工具构建结构化的经验单元，记录调用上下文、参数模式和结果。 + +2. **经验蒸馏（Experience Distillation）** + - 过滤无效 / 噪声经验，保留具有代表性的调用模式； + - 基于“等价类”视角对经验进行聚合，覆盖常见模式与稀有失败模式； + - 使用 LLM 对经验进行总结，形成可泛化的文本化指导（guidance）。 + +3. **经验复用（Experience Reuse）** + - 在未来任务中，根据当前工具调用上下文检索相关经验 / 指导； + - 将经验引导融入到工具选择、参数生成和响应整理等环节； + - 使得代理在面对动态环境和不完美反馈时仍能保持稳定表现。 + +--- + +### 主实验结果 + +MetaTool、API-Bank、BFCL-V3 上的性能对比（%）。**加粗**为各模型组内最优。 + +| Model | Method | MetaTool Pass@1 | MetaTool Avg@3 | MetaTool Pass@3 | API-Bank Pass@1 | API-Bank Avg@3 | API-Bank Pass@3 | BFCL-V3 Pass@1 | BFCL-V3 Avg@3 | BFCL-V3 Pass@3 | Total Pass@1 | Total Avg@3 | Total Pass@3 | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| GPT-5 nano | No Method | 72.62 | 72.76 | 78.49 | 82.96 | 83.46 | 86.97 | 53.80 | 53.00 | 60.95 | 70.82 | 70.62 | 76.63 | +| GPT-5 nano | Few-shot | 74.12 | 75.11 | 82.32 | 83.71 | 83.96 | **87.22** | 56.18 | 55.24 | 61.39 | 72.36 | 72.65 | 79.28 | +| GPT-5 nano | DRAFT | 73.94 | 73.04 | 78.97 | 84.21 | 83.46 | **87.22** | 57.27 | 57.27 | 62.26 | 72.52 | 71.58 | 77.23 | +| GPT-5 nano | Mem0 | 74.96 | 76.13 | 82.92 | 84.96 | 85.21 | **87.22** | 60.95 | 61.61 | 65.08 | 73.98 | 74.67 | 80.35 | +| GPT-5 nano | **ExpG** | **81.67** | **82.07** | **84.60** | **86.72** | **86.55** | **87.22** | **64.43** | **63.99** | **66.38** | **79.32** | **79.22** | **81.69** | +| DeepSeek-V3 | No Method | 83.10 | 82.94 | 84.66 | 84.71 | 84.38 | 85.46 | 58.79 | 59.65 | 65.94 | 78.92 | 78.66 | 81.37 | +| DeepSeek-V3 | Few-shot | 82.74 | 83.90 | 86.28 | 85.21 | 84.63 | 86.22 | 60.52 | 60.30 | 67.90 | 79.08 | 79.45 | 82.92 | +| DeepSeek-V3 | DRAFT | 80.23 | 80.79 | 82.44 | 84.96 | 85.63 | 86.47 | 62.26 | 61.61 | 68.55 | 77.70 | 77.80 | 80.54 | +| DeepSeek-V3 | Mem0 | 83.88 | 84.56 | 86.40 | 85.46 | 85.55 | 86.47 | 65.08 | 65.15 | 68.33 | 80.70 | 80.91 | 83.12 | +| DeepSeek-V3 | **ExpG** | **85.26** | **85.38** | **86.52** | **87.72** | **87.39** | **87.97** | **69.41** | **69.92** | **72.02** | **82.76** | **82.61** | **84.11** | +| Qwen3-8B | No Method | 76.51 | 76.97 | 77.71 | 83.96 | 83.88 | 84.21 | 58.79 | 58.28 | 60.30 | 74.46 | 74.41 | 75.56 | +| Qwen3-8B | Few-shot | 79.93 | 79.83 | 82.92 | 83.71 | 82.62 | 84.96 | 60.09 | 59.29 | 61.39 | 76.91 | 76.27 | 79.32 | +| Qwen3-8B | DRAFT | 78.19 | 77.33 | 77.89 | 85.71 | 84.96 | 85.46 | 60.74 | 60.30 | 62.91 | 76.20 | 75.18 | 76.35 | +| Qwen3-8B | Mem0 | 75.07 | 75.47 | 82.38 | 86.22 | 86.05 | 86.47 | 63.34 | 64.93 | 66.16 | 74.69 | 74.98 | 80.07 | +| Qwen3-8B | **ExpG** | **83.52** | **84.88** | **85.08** | **86.47** | **87.89** | **87.97** | **67.46** | **66.96** | **68.33** | **81.06** | **81.82** | **82.48** | +| Qwen3-32B | No Method | 80.05 | 79.43 | 80.17 | 84.71 | 84.88 | 85.21 | 65.15 | 65.08 | 66.16 | 78.05 | 77.55 | 78.41 | +| Qwen3-32B | **ExpG** | **84.68** | **85.02** | **86.28** | **86.97** | **87.30** | **87.72** | **70.72** | **71.01** | **73.32** | **82.48** | **82.56** | **84.14** | +| Qwen3-235B | No Method | 78.25 | 79.23 | 80.29 | 85.46 | 85.46 | 85.71 | 71.37 | 71.15 | 73.54 | 78.13 | 78.49 | 79.91 | +| Qwen3-235B | **ExpG** | **86.34** | **86.70** | **86.94** | **87.47** | **86.97** | **88.22** | **79.61** | **78.52** | **80.04** | **85.29** | **84.98** | **85.69** | + +--- + +### 参考代码 + +| 路径 | 作用 | +| --- | --- | +| [`tool_memory.py`](./tool_memory.py) | 官方风格 ReMe Tool Memory HTTP 客户端（`add_tool_call_result` / `summary_tool_memory` / `retrieve_tool_memory`） | +| [`parse_tool_call_result_prompt.yaml`](./parse_tool_call_result_prompt.yaml) | 单次工具调用多维评估用的 prompt | +| [`summary_tool_memory_prompt.yaml`](./summary_tool_memory_prompt.yaml) | 将工具调用历史总结为 guidance 的 prompt | +| [`tool_memory_flows.yaml`](./tool_memory_flows.yaml) | Tool Memory 相关的 flow / op 配置摘录 | + +以上为参考片段。完整可运行代码见 [WangCan1178/ExpG](https://github.com/WangCan1178/ExpG)。 + +--- + +### 引用 + +```bibtex +@misc{wang2026expg, + title = {Towards Robust Tool Use in Agents via Experience-Driven Adaptive Guidance}, + author = {Can Wang and Haoran Chen and Li Yu and Ding Hao and Bohai Zhao and Zhaoyang Liu and Zhiying Tu}, + year = {2026}, + eprint = {2608.03403}, + archivePrefix = {arXiv}, + primaryClass = {cs.AI}, + url = {https://arxiv.org/abs/2608.03403}, + howpublished = {\url{https://github.com/WangCan1178/ExpG}} +} +``` diff --git a/benchmark/toolmemory/gitcha.png b/benchmark/toolmemory/gitcha.png new file mode 100644 index 00000000..bb1152ec Binary files /dev/null and b/benchmark/toolmemory/gitcha.png differ diff --git a/benchmark/toolmemory/parse_tool_call_result_prompt.yaml b/benchmark/toolmemory/parse_tool_call_result_prompt.yaml new file mode 100644 index 00000000..371a12aa --- /dev/null +++ b/benchmark/toolmemory/parse_tool_call_result_prompt.yaml @@ -0,0 +1,49 @@ +prompt: | + You are an expert in evaluating tool invocation process. The tool is invoked by an AI agent. + + Tool invocation Information: + - Tool Name: {tool_name} + - Success Flag: {success_flag} + - Time Cost: {time_cost}s + - Token Cost: {token_cost} tokens + - Agent Context: {context} + - Input Parameters: {input_params} + - Tool Response: {response} + - Tool Schema: {schema} + + Evaluation Method: + Start from a default score list of scores = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]. + For each item below that is satisfied, assign 1 point to the corresponding index. + The final scores should be a list of 10 integers, each being either 0 or 1. + + 1. Use Quality (total 2 points. If context is provided, use it as an aid when evaluating): + - Index 1: Should the tool be invoked now? Consider whether all necessary information for the tool's invocation is ready, and whether the tool execution environment is correct. If it is a multi-round conversation, also consider the dependency relationships of the tool chain. + - Index 2: If should, is the chosen tool appropriate? + + 2. Input Quality (total 4 points. When evaluating, consider both the context and the tool schema): + - Index 3: Are all required parameters provided? + - Index 4: Are the input parameters valid and supported by the tool? + - Index 5: Are the input parameters in the correct format for their respective fields? + - Index 6: Does the value (content) of input parameter correctly reflect and match the given context? + + 3. Response Quality (total 4 points): + - Index 7: Does the response provide meaningful and useful information? Or are there any error messages or information that can be used as guidance for agent invoking tool better? + - Index 8: Does the response match the tool's intended purpose/function? + - Index 9: Does the response value correct (content appropriate) given the input parameters? + - Index 10: Does the response help accomplish the task within the given context? + + Important: + 1. Sometimes there is not enough information in the context or schema to make a complete evaluation. In such cases, make your best judgment based on the available information. + 2. Some tools (commonly system tools such as mkdir, touch, echo, etc.) modify the external environment. Since these results cannot be obtained, they return "None" as the response. At this point, all the scores in the quality of the response should be obtained and should not be seen as a problem for the tool. + 3. Evaluation independently from the success flag. The success_flag indicates whether the tool executed without technical errors. The evaluation should evaluate the quality of the tool invocation. A tool can execute successfully (Success Flag=1) but still produce low-quality or irrelevant responses, leading to a low evaluation score. + 4. Sometimes an agent will execute multiple steps and invoke multiple tools to complete a task, but you only need to evaluate the use of one tool for one of the steps, not whether the final task is completed or not. + + Answer Format: + Please provide your answer in the following JSON format: + + ```json + { + "scores": [0,0,0,0,0,0,0,0,0,0], + "explanation": "A brief evaluation (2-3 sentences) explaining the quality of the tool invocation, based on your evaluation. Low-quality aspects need to be reified, especially the causes of tool invocation errors." + } + ``` diff --git a/benchmark/toolmemory/summary_tool_memory_prompt.yaml b/benchmark/toolmemory/summary_tool_memory_prompt.yaml new file mode 100644 index 00000000..bff805c1 --- /dev/null +++ b/benchmark/toolmemory/summary_tool_memory_prompt.yaml @@ -0,0 +1,32 @@ +prompt: | + You are an expert in analyzing tool usage patterns and generating practical usage guidance for agents. + + Tool Information: + - Tool Name: {tool_name} + - Tool Schema: {tool_schema} + + Recent Tool Invocation Experiences: + {experiences} + + Important: + 1. Assume the tool (tool schema) can't be changed, your task is to guide agent to use it better. + 2. Your answer must be based on the information given, don't make it up. If not enough data, state "Not enough data to determine Core Function/Success Patterns/Common Issues/Best Practices." + 3. Your answer will be used to guide the use of the tool in the future, so do not include content related to recent tool invocation experience such as "case #3" or "Call #2", but some values can be used as examples. + 4. Pay attention to information not mentioned in the tool schema, such as the response upon successful tool invocation. It's also welcome to uncover insights, such as how tools can be used more effectively, and possible dependencies between tools. But if they aren't, don't make them up. + 5. Finally, to avoid deriving incorrect guidance from individual invocation, check whether, if the agent follows the proposed guidance, it can perform better on all recent invocation histories. If not, revise the guidance until it can. Specifically: + - Don't write guidance in an absolute tone without a very deterministic message (meaning that all invocation histories are satisfied, otherwise it will result in failure). + - Sometimes there may be inconsistencies. Consider whether this is due to the context in which the tool is being used. + + Your Task: + Based on the tool invocation history, generate a concise and logical tool usage guidance following this structure: + 1. Core Function: What this tool does and when to use it. + 2. Success Patterns: Parameter patterns and usage scenarios that work well. + 3. Common Issues: Main pitfalls to avoid and why they fail. + 4. Best Practices: 2-3 actionable recommendations. + + Answer Format: + Provide a structured, concise guidance (max 200 words). Focus on actionable insights derived from actual usage data. Avoid generic advice and think step by step. + + ```txt + Your concise, data-driven tool usage guidance + ``` \ No newline at end of file diff --git a/benchmark/toolmemory/tool_memory.py b/benchmark/toolmemory/tool_memory.py new file mode 100644 index 00000000..4a433de6 --- /dev/null +++ b/benchmark/toolmemory/tool_memory.py @@ -0,0 +1,234 @@ +"""Official-style ReMe Tool Memory HTTP helpers. + +Aligned with ReMe Tool Memory HTTP APIs (see ReMe cookbook +``use_tool_memory_demo.py`` and docs under ``docs/tool_memory/``): + +- ``add_tool_call_result`` +- ``summary_tool_memory`` +- ``retrieve_tool_memory`` + +Response memories are read from ``metadata.memory_list[].content``. +This module does not use ExpG-only fields such as ``no_persist``, +``source_task``, or ``add_to``. +""" + +from __future__ import annotations + +import logging +from typing import Any, Dict, List, Optional + +import httpx + +logger = logging.getLogger(__name__) + +DEFAULT_BASE_URL = "http://localhost:8002" + + +class ToolMemoryFetcher: + """HTTP client for ReMe Tool Memory endpoints.""" + + def __init__( + self, + workspace_id: str, + base_url: str = DEFAULT_BASE_URL, + timeout: float = 60.0, + ) -> None: + self.workspace_id = workspace_id + self.base_url = base_url.rstrip("/") + self.timeout = timeout + + def _url(self, endpoint: str) -> str: + return f"{self.base_url}/{endpoint.lstrip('/')}" + + @staticmethod + def _join_tool_names(tool_names: List[str] | str) -> str: + if isinstance(tool_names, str): + return tool_names + return ",".join(tool_names) + + @staticmethod + def _memory_list(payload: Dict[str, Any]) -> List[Dict[str, Any]]: + metadata = payload.get("metadata") or {} + if not isinstance(metadata, dict): + return [] + memory_list = metadata.get("memory_list") or [] + return memory_list if isinstance(memory_list, list) else [] + + @classmethod + def _content_by_tool(cls, payload: Dict[str, Any]) -> Dict[str, str]: + result: Dict[str, str] = {} + for memory in cls._memory_list(payload): + if not isinstance(memory, dict): + continue + tool_name = str(memory.get("when_to_use") or "").strip() + content = memory.get("content") or "" + if tool_name: + result[tool_name] = str(content) + return result + + async def add_tool_call_result_async( + self, + tool_call_results: List[Dict[str, Any]], + ) -> Dict[str, Any]: + """Call ``add_tool_call_result``.""" + async with httpx.AsyncClient() as client: + response = await client.post( + self._url("add_tool_call_result"), + json={ + "workspace_id": self.workspace_id, + "tool_call_results": tool_call_results, + }, + timeout=self.timeout, + ) + response.raise_for_status() + return response.json() + + async def summary_tool_memory_async( + self, + tool_names: List[str] | str, + ) -> Dict[str, Any]: + """Call ``summary_tool_memory``.""" + async with httpx.AsyncClient() as client: + response = await client.post( + self._url("summary_tool_memory"), + json={ + "workspace_id": self.workspace_id, + "tool_names": self._join_tool_names(tool_names), + }, + timeout=self.timeout, + ) + response.raise_for_status() + return response.json() + + async def retrieve_tool_memory_async( + self, + tool_names: List[str] | str, + ) -> Dict[str, Any]: + """Call ``retrieve_tool_memory``.""" + async with httpx.AsyncClient() as client: + response = await client.post( + self._url("retrieve_tool_memory"), + json={ + "workspace_id": self.workspace_id, + "tool_names": self._join_tool_names(tool_names), + }, + timeout=self.timeout, + ) + response.raise_for_status() + return response.json() + + async def collect_memory_async( + self, + tool_names: List[str], + ) -> Dict[str, str]: + """Summarize then retrieve guidance for tools. + + Returns: + Mapping from tool name to memory ``content`` string. + """ + if not tool_names: + return {} + + names = self._join_tool_names(tool_names) + try: + summary = await self.summary_tool_memory_async(names) + if not summary.get("success"): + logger.warning("summary_tool_memory failed for %s", names) + except Exception as exc: # noqa: BLE001 + logger.warning("summary_tool_memory error for %s: %s", names, exc) + + try: + retrieved = await self.retrieve_tool_memory_async(names) + except Exception as exc: # noqa: BLE001 + logger.warning("retrieve_tool_memory error for %s: %s", names, exc) + return {} + + if not retrieved.get("success"): + logger.warning("retrieve_tool_memory failed for %s", names) + return {} + + return self._content_by_tool(retrieved) + + def add_tool_call_result( + self, + tool_call_results: List[Dict[str, Any]], + ) -> Dict[str, Any]: + """Sync wrapper for ``add_tool_call_result``.""" + with httpx.Client() as client: + response = client.post( + self._url("add_tool_call_result"), + json={ + "workspace_id": self.workspace_id, + "tool_call_results": tool_call_results, + }, + timeout=self.timeout, + ) + response.raise_for_status() + return response.json() + + def summary_tool_memory(self, tool_names: List[str] | str) -> Dict[str, Any]: + """Sync wrapper for ``summary_tool_memory``.""" + with httpx.Client() as client: + response = client.post( + self._url("summary_tool_memory"), + json={ + "workspace_id": self.workspace_id, + "tool_names": self._join_tool_names(tool_names), + }, + timeout=self.timeout, + ) + response.raise_for_status() + return response.json() + + def retrieve_tool_memory(self, tool_names: List[str] | str) -> Dict[str, Any]: + """Sync wrapper for ``retrieve_tool_memory``.""" + with httpx.Client() as client: + response = client.post( + self._url("retrieve_tool_memory"), + json={ + "workspace_id": self.workspace_id, + "tool_names": self._join_tool_names(tool_names), + }, + timeout=self.timeout, + ) + response.raise_for_status() + return response.json() + + def collect_memory(self, tool_names: List[str]) -> Dict[str, str]: + """Sync wrapper for summarize + retrieve. + + Prefer ``collect_memory_async`` inside an existing event loop. + """ + if not tool_names: + return {} + + names = self._join_tool_names(tool_names) + try: + summary = self.summary_tool_memory(names) + if not summary.get("success"): + logger.warning("summary_tool_memory failed for %s", names) + except Exception as exc: # noqa: BLE001 + logger.warning("summary_tool_memory error for %s: %s", names, exc) + + try: + retrieved = self.retrieve_tool_memory(names) + except Exception as exc: # noqa: BLE001 + logger.warning("retrieve_tool_memory error for %s: %s", names, exc) + return {} + + if not retrieved.get("success"): + logger.warning("retrieve_tool_memory failed for %s", names) + return {} + + return self._content_by_tool(retrieved) + + def get_memory_content( + self, + tool_names: List[str] | str, + ) -> Optional[str]: + """Retrieve and join memory contents for the given tools.""" + payload = self.retrieve_tool_memory(tool_names) + if not payload.get("success"): + return None + contents = [content for content in self._content_by_tool(payload).values() if content] + return "\n\n".join(contents) if contents else None diff --git a/benchmark/toolmemory/tool_memory_flows.yaml b/benchmark/toolmemory/tool_memory_flows.yaml new file mode 100644 index 00000000..b62d0571 --- /dev/null +++ b/benchmark/toolmemory/tool_memory_flows.yaml @@ -0,0 +1,45 @@ +# Tool Memory flow / op config excerpt used by ExpG. +# Full runnable code: https://github.com/WangCan1178/ExpG + +flow: + retrieve_tool_memory: + flow_content: retrieve_tool_memory_op + description: "Retrieves tool memories from the vector database based on tool names to provide tool usage patterns and best practices" + input_schema: + tool_names: + type: string + description: "Comma-separated tool names (e.g., 'tool_name1,tool_name2')" + required: true + + add_tool_call_result: + flow_content: parse_tool_call_result_op >> update_vector_store_op + description: "Evaluates and adds tool call results to the tool memory database, creating new memory or updating existing memory for the specified tool" + input_schema: + tool_call_results: + type: array + description: "List of tool call result objects, each containing: tool_name, input, output, success, time_cost, token_cost, create_time" + required: true + + summary_tool_memory: + flow_content: summary_tool_memory_op >> update_vector_store_op + description: "Analyzes tool call history and generates comprehensive usage patterns, best practices, and recommendations for the specified tools" + input_schema: + tool_names: + type: string + description: "Comma-separated tool names to summarize (e.g., 'tool_name1,tool_name2')" + required: true + +op: + parse_tool_call_result_op: + backend: parse_tool_call_result_op + llm: default + params: + max_history_tool_call_cnt: 100 + evaluation_sleep_interval: 1.0 + + summary_tool_memory_op: + backend: summary_tool_memory_op + llm: default + params: + data_from: '2025-09-10 10:56:58' + summary_sleep_interval: 1.0