diff --git a/README.md b/README.md index 27acedf5..7b5e4061 100644 --- a/README.md +++ b/README.md @@ -66,15 +66,15 @@ working_dir/ [ReMeLight](reme/reme_light.py) is the core class of this memory system, providing complete memory management capabilities for AI Agents: -| Method | Function | Key Components | -|------------------------|------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------| -| `start` | 🚀 Start memory system | Initialize file store, file watcher, Embedding cache; clean up expired tool result files | -| `close` | 📕 Close and clean up | Clean tool result files, stop file watcher, save Embedding cache | -| `compact_memory` | 📦 Compact history to summary | [Compactor](reme/memory/file_based/compactor.py) — ReActAgent generates structured context checkpoint | -| `summary_memory` | 📝 Write important memory to files | [Summarizer](reme/memory/file_based/summarizer.py) — ReActAgent + file tools (read / write / edit) | -| `compact_tool_result` | ✂️ Compact oversized tool output | [ToolResultCompactor](reme/memory/file_based/tool_result_compactor.py) — Truncate and save to `tool_result/`, keep file reference in message | -| `pre_reasoning_hook` | 🔄 Pre-reasoning hook | Auto compact tool results + generate summary + async trigger memory summarization task | -| `memory_search` | 🔍 Semantic memory search | [MemorySearch](reme/memory/tools/chunk/memory_search.py) — Vector + BM25 hybrid retrieval | +| Method | Function | Key Components | +|------------------------|------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `start` | 🚀 Start memory system | Initialize file store, file watcher, Embedding cache; clean up expired tool result files | +| `close` | 📕 Close and clean up | Clean tool result files, stop file watcher, save Embedding cache | +| `compact_memory` | 📦 Compact history to summary | [Compactor](reme/memory/file_based/compactor.py) — ReActAgent generates structured context checkpoint | +| `summary_memory` | 📝 Write important memory to files | [Summarizer](reme/memory/file_based/summarizer.py) — ReActAgent + file tools (read / write / edit) | +| `compact_tool_result` | ✂️ Compact oversized tool output | [ToolResultCompactor](reme/memory/file_based/tool_result_compactor.py) — Truncate and save to `tool_result/`, keep file reference in message | +| `pre_reasoning_hook` | 🔄 Pre-reasoning hook | Auto compact tool results + generate summary + async trigger memory summarization task | +| `memory_search` | 🔍 Semantic memory search | [MemorySearch](reme/memory/tools/chunk/memory_search.py) — Vector + BM25 hybrid retrieval | | `get_in_memory_memory` | 🗂️ Create in-memory instance | [ReMeInMemoryMemory](reme/memory/file_based/reme_in_memory_memory.py) — Token-aware memory management, supports compression summary and state serialization (static method) | --- @@ -125,9 +125,9 @@ async def main(): summary = await reme.compact_memory( messages=messages, previous_summary="", - max_input_length=128000, # Model context window (tokens) - compact_ratio=0.7, # Trigger compaction when reaching max_input_length * 0.7 - language="zh", # Summary language (zh / "") + max_input_length=128000, # Model context window (tokens) + compact_ratio=0.7, # Trigger compaction when reaching max_input_length * 0.7 + language="zh", # Summary language (zh / "") ) # 3. Submit async summary task in background (non-blocking, writes to memory/YYYY-MM-DD.md) @@ -169,7 +169,8 @@ if __name__ == "__main__": ``` > 📂 Full example code: [test_reme_light.py](tests/light/test_reme_light.py) -> 📋 Example output: [test_reme_light.log](tests/light/test_reme_light.log) (223,838 tokens → 1,105 tokens, 99.5% compression ratio) +> 📋 Example output: [test_reme_light.log](tests/light/test_reme_light_log.txt) (223,838 tokens → 1,105 tokens, 99.5% +> compression ratio) ### File-Based ReMeLight Memory System Architecture diff --git a/README_ZH.md b/README_ZH.md index ded1d8d1..3c379a2b 100644 --- a/README_ZH.md +++ b/README_ZH.md @@ -97,7 +97,6 @@ pip install -e ".[light]" ```python import asyncio -from agentscope.message import Msg from reme.reme_light import ReMeLight @@ -119,9 +118,9 @@ async def main(): summary = await reme.compact_memory( messages=messages, previous_summary="", - max_input_length=128000, # 模型上下文窗口(tokens) - compact_ratio=0.7, # 达到 max_input_length * 0.7 时触发压缩 - language="zh", # 摘要语言(zh / "") + max_input_length=128000, # 模型上下文窗口(tokens) + compact_ratio=0.7, # 达到 max_input_length * 0.7 时触发压缩 + language="zh", # 摘要语言(zh / "") ) # 3. 后台异步提交摘要任务(不阻塞对话,摘要写入 memory/YYYY-MM-DD.md) @@ -163,7 +162,7 @@ if __name__ == "__main__": ``` > 📂 完整示例代码:[test_reme_light.py](tests/light/test_reme_light.py) -> 📋 运行结果示例:[test_reme_light.log](tests/light/test_reme_light.log)(223,838 tokens → 1,105 tokens,压缩率 99.5%) +> 📋 运行结果示例:[test_reme_light.log](tests/light/test_reme_light_log.txt)(223,838 tokens → 1,105 tokens,压缩率 99.5%) ### 基于文件的 ReMeLight 记忆系统架构 diff --git a/tests/light/test_reme_light_log.txt b/tests/light/test_reme_light_log.txt new file mode 100644 index 00000000..5a0c71a5 --- /dev/null +++ b/tests/light/test_reme_light_log.txt @@ -0,0 +1,84 @@ +====================================================================== +ReMeLight 已启动 +====================================================================== + +[原始消息]: 18 条, 223,838 tokens + 目标阈值: 128K = 131,072 tokens + 超出阈值: True + +====================================================================== +[步骤 1] compact_tool_result - 压缩超长工具输出 +====================================================================== + 消息数量: 18 → 18 + 📊 Token 统计: 223,838 → 1,107 (变化: -222,731, -99.5%) + +====================================================================== +[步骤 2] compact_memory - 生成结构化压缩摘要 +====================================================================== + 输入消息 tokens: 223,838 + 压缩摘要长度: 1032 字符, 493 tokens + 压缩比: 0.2% + +====================================================================== +[步骤 3] summary_memory - 生成完整摘要并写入文件 +====================================================================== + +reme_summarizer: [SILENT] + 输入消息 tokens: 223,838 + 摘要结果长度: 8 字符 + 摘要: [SILENT] + +====================================================================== +[步骤 4] pre_reasoning_hook - 推理前预处理 +====================================================================== + 消息数量: 18 → 18 + 📊 Token 统计: 223,838 → 1,105 (变化: -222,733, -99.5%) + 压缩摘要: 0 字符, 0 tokens + 总上下文: 1,105 tokens + +====================================================================== +[步骤 5] memory_search - 语义搜索记忆 +====================================================================== + 搜索结果: [{'type': 'text', 'text': '[\n {\n "path": "/Users... + +====================================================================== +[步骤 6] ReMeInMemoryMemory - 会话内存管理 +====================================================================== + 已添加 18 条原始消息到内存 + +[6.1] estimate_tokens - 估算 Token 使用: + - 总消息数: 18 + - 消息 Token 数: 223,838 + - 压缩摘要 Token 数: 0 + - 预估总 Token 数: 223,838 + - 最大输入长度: 128,000 + - 上下文使用率: 174.87% + +[6.2] get_history_str - 格式化历史记录: +**Conversation History** + +- Total messages: 18 +- Estimated tokens: 223838 +- Max input length: 128000 +- Context usage: 174.9% +- Compressed summary tokens: 0 +... + +====================================================================== +[步骤 7] 等待后台任务完成 +====================================================================== + 后台任务完成,结果长度: 0 字符 + +====================================================================== +📊 Token 变化总结 +====================================================================== + 原始消息: 223,838 tokens + Step 1 compact_tool_result 后: 1,107 tokens + Step 2 compact_memory 摘要: 493 tokens + Step 4 pre_reasoning_hook 后: 1,105 tokens + 摘要 0 tokens = 1,105 tokens + 最大节省: 222,733 tokens (99.5%) + 目标阈值: 131,072 tokens + +====================================================================== +ReMeLight 已关闭 +======================================================================