mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-09-07 08:26:06 +00:00
3 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
jinliyl
|
83831ec90c
|
feat(core): enhance reme4 (#281)
### 1. Agent Wrapper（统一 Agent 后端抽象） - **`base_agent_wrapper.py`**：`reply()` 返回值从 `tuple[str, Any]` 改为 `dict`（含 `session_id` / `last_message` / `result` / 可选 `structured_output`）；`reply_stream()` 改为产出统一的 `StreamChunk`。废弃 `add_tools()`，改为 `add_job_tools(names: list[str])`（按名解析 BaseJob）与 `add_skills()`；新增 `_resolve_job_tools()`、`_merged_kwargs()`、`_chunk()` 辅助方法及 `project_path` / `project_skills_root` 属性。 - **`as_agent_wrapper.py`（AgentScope 后端）**： - 会话持久化重写：`session_path` 落地到 `<vault>/<session_dir>/agentscope/`，`_load_state` 支持 `resume` / `session_id` / `fork_session`，并做 UUID 校验（`_validate_session_id`）；`_cleanup_expired_sessions` 按天数清理过期会话。 - 新增内置工具集（`BypassAnalysisBash` + Edit/Glob/Grep/Read/Write），`BypassAnalysisBash` 绕过 AgentScope 自带 Bash 静态分析以让 permission_mode 生效；`_resolve_skills()` 把配置的 skill 暴露给后端，`_load_tool_env()` 注入项目 `.env`。 - `_event_to_chunk()` 把 20+ 种 AgentScope 事件（Reply/Text/Thinking/Data/ToolCall/ToolResult/ModelCall/ExceedMaxIters）归一化为 `StreamChunk`。 - **`cc_agent_wrapper.py`（Claude Code SDK 后端，+551 行）**： - 新增 `_CcFileSessionStore`：基于 vault 的文件型会话存储，实现 append（按 uuid 去重）/ load / list / delete / list_subkeys，并对路径做 `_safe_parts` + `resolve()` 防越界校验。 - `_build_options()`：统一构建 `ClaudeAgentOptions`，处理 skills、disallowed_tools（默认禁 `WebSearch`）、`.env` 注入、Claude Code 的 API 凭据解析（`_claude_code_api_env`，多级 base_url/api_key 回退）、`CLAUDE_CONFIG_DIR` 设置、skill 目录软链接（`_ensure_claude_skill_dir`）。 - `_raw_event_to_chunk()` / `_message_content_to_chunks()`：把 Anthropic 流式事件（message_start/delta/stop、content_block_*）与 SDK 消息块（AssistantMessage/UserMessage/ResultMessage/RateLimitEvent）转换为统一 `StreamChunk`；跟踪 block_id/block_type/tool_call_name 做关联；处理尾部 `"success"` 误报异常的吞掉逻辑。 ### 2. 统一流式协议（StreamChunk / ChunkEnum） - **`stream_chunk.py`**：`StreamChunk` 扩展为承载 AS + CC 双后端完整信息的统一结构，新增 `session_id` / `block_id` / `tool_call_id` / `tool_call_name` / `media_type` / `input_tokens` / `output_tokens` 等字段，纯文本流仍保持轻量。 - **`chunk_enum.py`**：补全生命周期标记 `REPLY_START` / `REPLY_END`，并文档化两套后端事件 → ChunkEnum 的映射。 ### 3. Index 模块重构（变化批次化 + dispatch） - 新增 `_change_batch.py`：`coalesce_changes()` 把同路径多次事件折叠为最终状态（结合 path 存在性判定），`bucket_changes()` 按 watchfiles.Change 分桶。 - 新增 `init_changes.py`（`InitChangesStep`）：一次性扫描，对比 file_store / file_catalog 已索引节点计算 added/modified/deleted，写入 `context["changes"]` 后 dispatch。 - 新增 `update_changes.py`：抽象基类 `ChangeApplyStep` 统一 added/modified/deleted 处理与错误收集；`UpdateCatalogStep`（写 file_catalog）、`UpdateIndexStep`（写 file_store，含按后缀解析 chunker）。 - **`watch_changes.py`**：改用 `dispatch_step_specs`（基类提供的 `dispatch_steps()`），每批先 `coalesce_changes` 再 dispatch；默认参数调整（debounce 5000ms / step 1000ms / poll 5000ms）并暴露常量。 - 删除旧步骤：`clear_and_scan` / `foreach_dispatch` / `scan_changes` / `update_catalog`(旧) / `update_index`(旧)；`clear_store.py` 取代 clear_and_scan。 ### 4. Evolve / Dream 模块（拆分为多步 pipeline） - 删除旧的单体 `auto_dream.py` / `dream.py` / `dream.yaml`，新增 `dream/` 子包，按 5 个步骤组织： - **`extract.py`**：扫描当日 day-index + daily 笔记，对比 file_catalog 找出 changed/deleted，调用 LLM 全局抽取 `units`（procedure/personal/wiki 三桶）与 `topics`，路径与桶做清洗/路由。 - **`integrate.py`**：逐个 unit 调用 LLM 写入 digest，结构化输出 `IntegrateOutcome`（CREATE/CORROBORATE/REFINE/CORRECT），失败 unit/路径收集回写。 - **`topics.py`**：写 `daily/<date>/interests.yaml`，结合当天已有 + 近 N 天做去重（`normalize_topic`），可走 LLM 或纯规则去重两条路径。 - **`proactive.py`**：读取当日 `interests.yaml`，作为主动推荐话题的入口。 - **`finish.py`**：把变更路径落盘到 dream file_catalog（checkpoint），渲染最终汇总摘要。 - 新增 `schema.py`（`DreamState` 等跨步骤共享状态与结构化输出模型）与 `utils.py`（状态存取、扫描打包、YAML 读写、结构化回复解析等公共函数）。 - `evolve/__init__.py` 导出全部新 step。 ### 5. auto_memory / auto_resource（适配新 Agent API） - **`auto_memory.py`**：会话路径迁移到 `<session_dir>/dialog/<session_id>.jsonl`；改用 `job_tools`；新增 `source_conversation` frontmatter 反向链接（`_session_link`）；执行后刷新 day 索引（`refresh_day_index`），并对 session_id 做合法性校验。 - **`auto_resource.py`**：资源改用「同名 daily note」方案（`_compute_note_stem` 取文件 stem）；批量处理 `changes: list[dict]`（`_handle_change` 逐项处理，返回逐项结果摘要）；agent 会话 id 用稳定的 `uuid5`；同样刷新 day 索引。 ### 6. BaseStep 基类增强 - 新增 `dispatch_steps` / `dispatch_step_specs` 机制：`_resolve_dispatch_step()` 支持字符串或 dict 形式的 step spec，`dispatch_steps()` 复用当前 context 调用下游 step。 - 新增 `config_value()`：按 key 取 app config，缺失时回退 `ApplicationConfig` 默认值。 - 小幅清理：`language` 初始化、`copy()`、`Ref.__init__` 签名精简。 ### 7. Components 改动 - **`file_store/local_file_store.py`**：持久化改用 zstd 压缩（`.jsonl.zst`，通过新 `utils/jsonl_zst.py`）；upsert 时先删除旧 chunk 的 keyword 文档；embedding 复用改为 `(text, embedding)` 键控，要求文本一致才复用；新增 `_matches_search_filter()` 对 vector/keyword 搜索做 path/path_prefix/metadata 的统一后过滤。 - **`keyword_index/bm25_index.py`**：索引文件名加入组件名 + tokenizer 指纹（sha256 前 12 位），快照/恢复时校验指纹防配置漂移；空索引 dump 时删除文件，加载失败抛错而非静默。 - **`file_chunker/markdown_file_chunker.py`**：弃用 `python-frontmatter`，改用内置 YAML 解析（非法 YAML 不阻断正文索引），并修正因 frontmatter 占用行号导致的 AST 行号偏移（`line_offset`）。 - **`cron_job.py`**：大幅简化（-187 行），由原来「dispatch 外部 job/step + 多种调度模式」改为「在自身 steps 上跑 cron 表达式」；`Application` 启动顺序随之调整为 base > stream > background > cron。 - 其余小调整：service（base/http/mcp）、file_graph、file_catalog、as_llm、as_embedding、tokenizer、prompt_handler、base_component 的签名/接口微调。 ### 8. Application 生命周期 - `_start()` 启动顺序明确为 components → base → stream → background → cron，启动失败会触发 `_close()` 回滚并 re-raise（不再吞异常）。 - 启动时创建 `session_dir` 目录；新增 `update_component()`（按类型/名就地更新已存在组件，不存在则报错）。 ### 9. File IO / 路径安全 - **`_path.py`**：`resolve_path` 增加 vault 越界防护（`is_relative_to` 校验），禁止 `.` / `..` 路径分量，支持 `allow_empty`。 - **`read.py`**：大文件（超过 `MAX_FILE_READ_BYTES`）走按行读取 `read_file_lines_safe`，避免一次性载入内存。 - **`_file_io.py` / `_daily_index.py` / `_path.py`** 等支持函数补齐（如 `refresh_day_index`、`read_file_lines_safe`）。 - **`env_utils.py`**：新增 `parse_env_file()`，`load_env()` 返回加载到的键值、支持 `override`、对无路径调用做幂等缓存。 ### 10. Config - `ApplicationConfig` 新增 `session_dir`（默认 `reme_session`）。 - `config_parser.py`：环境变量展开后做类型转换（`_convert_value`）、dot-notation 与 key=value 参数校验更严格、配置文件路径支持相对 `_CONFIG_DIR` 查找、根非 dict 报错。 - `default.yaml`：作业编排改用 `init_changes_step` + `dispatch_steps`（index/resource/digest 三个 watch loop 与 reindex）；新增 `auto_dream`（4 步）、`proactive` 作业，移除旧 `dream`；file_catalog 增配 `resource` / `digest` / `dream` 实例；LLM 默认值与 Claude Code 凭据配置调整（tool_result_limit 50000、thinking_enable=false 等）。 ### 11. 其它 - 新增 `steps/common/add.py`（`AddStep` 算术 demo）、`channel/__init__.py` 与 common `__init__` 导出整理。 - 新增 4 篇文档：`docs4/auto_dream_logic_and_step_refactor.md`、`docs4/watch_loop_step_refactor_plan.md`、`docs4/todo.md`，以及 `reme_design.md` 更新。 ** |
||
|
jinliyl
|
8eaa96390a
|
refactor(file_chunker): replace file parser with file chunker component (#276)
* refactor(file_chunker): replace file parser with file chunker component - Rename file_parser module to file_chunker across codebase - Update BaseFileParser to BaseFileChunker with corresponding component type - Rename LinkedFileParser to MarkdownFileChunker for markdown-specific chunking - Rename ChunkedFileParser to DefaultFileChunker for default byte-based chunking - Update documentation references from file_parser to file_chunker - Modify dependency injection in BaseStep to use file_chunker instead of file_parser - Update configuration and component registration to use new chunker naming - Rename all related test files and update test assertions accordingly - Add recursive option to scan_store_changes_step in default configuration * feat(database): enhance Neo4j connection with environment variable support - Add support for NEO4J_PASSWORD environment variable as fallback - Make password parameter optional in constructor with validation - Update chromadb dependency from 1.3.5 to 1.5.7 - Configure CORS credentials based on origin settings - Import os module for environment variable access * feat(config): add timezone support and remove unused dialog directory - Added timezone field to application config with IANA timezone support - Removed unused dialog_dir configuration and related directory creation - Replaced date.today() with timezone-aware now() function across daily operations - Created evolve module with timezone-aware datetime functionality - Updated daily_create, daily_list, and daily_reindex steps to use timezone-aware dates * refactor(steps): update file chunker implementation - Replace ChunkedFileParser with DefaultFileChunker in background steps - Add module docstring to evolve steps package - Update return type annotation to reflect new chunker class usage * refactor(components): rename embedding and llm components to as_embedding and as_llm - Rename reme4/components/embedding to reme4/components/as_embedding - Rename reme4/components/llm to reme4/components/as_llm - Update all imports and references from embedding to as_embedding - Update all imports and references from llm to as_llm - Change BaseEmbedding to BaseAsEmbedding and update inheritance - Change BaseLLM to BaseAsLLM and update inheritance - Update component types from LLM/EMBEDDING to AS_LLM/AS_EMBEDDING - Update configuration keys from embedding/llm to as_embedding/as_llm - Update all property references from llm to as_llm in step classes - Update test assertions to use new component enum values * refactor(embedding_store): rename embedding parameter to as_embedding - Updated configuration key from 'embedding' to 'as_embedding' - Renamed class attribute from 'embedding' to 'as_embedding' - Updated method calls to use 'as_embedding' instead of 'embedding' - Changed parameter name in constructor from 'embedding' to 'as_embedding' - Updated documentation to reflect new parameter name - Modified health check to use 'as_embedding' property * feat(agent_wrapper): add unified agent wrapper component with multiple backends - Introduce BaseAgentWrapper abstract base class for agent implementations - Add AsAgentWrapper implementation using AgentScope framework - Add CcAgentWrapper implementation using Claude Code SDK - Register agent_wrapper component type in ComponentEnum - Configure default agent_wrapper settings in default.yaml - Implement tool integration for both AgentScope and Claude Code backends - Support fluent configuration via set_system_prompt() and add_tools() methods * feat(agent-wrapper): add structured output support for agent wrappers - Import SystemMsg in AsAgentWrapper for structured output handling - Add output_schema parameter support in AsAgentWrapper with generate_structured_output - Implement set_output_schema method in BaseAgentWrapper for chaining configuration - Add output schema support in CcAgentWrapper with JSON schema format option - Return structured output when available in CcAgentWrapper response - Refactor kwargs handling to use default values consistently across wrapper classes |
||
|
jinliyl
|
d8086039dc
|
refactor(Agentscope2.0): llm & embedding & agent (#271)
* refactor(embedding): replace embedding model with embedding store architecture - Remove as_token_counter component and its estimated token counter implementation - Replace BaseEmbeddingModel with BaseEmbedding that wraps AgentScope embedding models - Add support for multiple embedding providers (OpenAI, DashScope, Gemini, Ollama) - Introduce BaseEmbeddingStore and LocalEmbeddingStore for caching and persistence - Update component registry to use new embedding and embedding_store types - Modify file stores to use embedding_store instead of embedding_model - Update health check to monitor embedding_store instead of embedding_model - Change default config to use embedding_store with local backend - Add estimate_token_count utility function to utils module * refactor(llm): replace as_llm components with unified llm implementation - Remove deprecated as_llm and as_llm_formatter modules - Add new llm module with BaseLLM and provider-specific implementations - Update component registry to use LLM instead of AS_LLM - Replace all as_llm/as_llm_formatter references with llm in steps - Update configuration schema to use llm instead of as_llm - Rename integration test file from test_as_llm to test_llm - Add proper docstrings to embedding store dimension property - Add pylint disable comment for embedding model call - Remove unused FormatterBase import in base_step - Update token_utils with function docstring * refactor(evolve): replace ReActAgent with Agent and update message handling - Removed FlexReActAgent class and direct ReActAgent imports - Updated Agent instantiation to use new constructor parameters - Changed message content to use TextBlock format instead of plain strings - Modified timestamp access from msg.timestamp to msg.created_at - Updated metadata access pattern for structured outputs - Replaced Msg.from_dict with Msg.model_validate in auto_memory.py - Updated test mocks to patch Agent instead of ReActAgent - Changed message serialization from to_dict to model_dump in tests - Moved component references to base class definition - Updated demo tools to return strings instead of ToolResponse objects * feat(step): migrate to FunctionTool and add streaming support - Replace deprecated ToolResponse with FunctionTool in base_step.py - Remove unused TextBlock import from base_step.py - Update job registration to use new FunctionTool API - Add thinking_budget parameter to llm_demo configuration - Introduce StreamLLMDemoStep with streaming output capability - Add structured output support to LLMDemoStep via generate_structured_output - Implement streaming event handling for text/thinking/tool calls - Add integration tests for embedding functionality - Add integration tests for structured output and streaming features - Update tool usage in demo steps to use new function naming convention * fix(ci): correct package installation path in unittest workflow - Updated pip install command to use proper package path "./reme4[dev,core]" - Fixed dependency installation step in CI workflow configuration * chore(workflow): update python versions in unittest workflow - Remove Python 3.10 from test matrix - Add Python 3.11 to test matrix - Add Python 3.12 to test matrix - Keep Python 3.13 in test matrix - Update matrix configuration for better version coverage * fix(health): handle missing dimensions attribute in embedding status - Wrap dimensions access in try-except to prevent AttributeError - Return None when dimensions attribute is not available - Maintain backward compatibility for components without dimensions test(component): add comprehensive tests for BaseComponent and related classes - Add tests for Dependency class including repr and attribute access - Add tests for bind method with various scenarios and edge cases - Add tests for lifecycle management and async context handling - Add tests for standalone and context-bound dependency resolution - Add tests for ComponentMixin path utilities test(common): update LocalFileStore initialization parameter - Change embedding_model parameter to embedding_store in test setup - Update all affected test files consistently test(registry): add complete test suite for ComponentRegistry - Add tests for register method with explicit names and defaults - Add tests for decorator registration pattern - Add tests for get_all method returning copies - Add tests for unregister and clear operations - Add tests for error handling of invalid registrations test(job): add comprehensive tests for BaseJob and BackgroundJob - Add tests for step resolution and exception handling - Add tests for backoff delay calculation with jitter - Add tests for supervisor loop restart behavior - Add tests for task shutdown and cancellation test(prompt): add complete test suite for PromptHandler - Add tests for prompt loading from dictionaries and files - Add tests for internationalization and language fallback - Add tests for flag filtering and variable substitution - Add tests for format validation and error handling test(runtime): add basic tests for RuntimeContext dictionary access - Add tests for item getting, setting and containment checks - Add tests for missing key error handling * feat(evolve): add permission context and agent state management - Import PermissionContext, PermissionMode and AgentState modules - Add state configuration with bypass permission mode to AutoDream agents - Add state configuration with bypass permission mode to AutoMemory agents - Implement static _to_msg method for message validation and formatting - Refactor message processing to use the new _to_msg method - Ensure proper content structure for text blocks in message conversion * style(tests): update test files with linting rules and code improvements - Add missing pylint disable directives for docstring and attribute warnings - Replace lambda expressions with proper function definitions in test cases - Import Path directly instead of using lambda with __import__ - Simplify assertion checks by using truthiness instead of equality to empty dict - Remove unused imports and reorder imports consistently - Format dictionary literals with proper indentation and line breaks |