27 commits

Author	SHA1	Message	Date
jinliyl	83831ec90c	feat(core): enhance reme4 (#281) ... ### 1. Agent Wrapper（统一 Agent 后端抽象） - **`base_agent_wrapper.py`**：`reply()` 返回值从 `tuple[str, Any]` 改为 `dict`（含 `session_id` / `last_message` / `result` / 可选 `structured_output`）；`reply_stream()` 改为产出统一的 `StreamChunk`。废弃 `add_tools()`，改为 `add_job_tools(names: list[str])`（按名解析 BaseJob）与 `add_skills()`；新增 `_resolve_job_tools()`、`_merged_kwargs()`、`_chunk()` 辅助方法及 `project_path` / `project_skills_root` 属性。 - **`as_agent_wrapper.py`（AgentScope 后端）**： - 会话持久化重写：`session_path` 落地到 `<vault>/<session_dir>/agentscope/`，`_load_state` 支持 `resume` / `session_id` / `fork_session`，并做 UUID 校验（`_validate_session_id`）；`_cleanup_expired_sessions` 按天数清理过期会话。 - 新增内置工具集（`BypassAnalysisBash` + Edit/Glob/Grep/Read/Write），`BypassAnalysisBash` 绕过 AgentScope 自带 Bash 静态分析以让 permission_mode 生效；`_resolve_skills()` 把配置的 skill 暴露给后端，`_load_tool_env()` 注入项目 `.env`。 - `_event_to_chunk()` 把 20+ 种 AgentScope 事件（Reply/Text/Thinking/Data/ToolCall/ToolResult/ModelCall/ExceedMaxIters）归一化为 `StreamChunk`。 - **`cc_agent_wrapper.py`（Claude Code SDK 后端，+551 行）**： - 新增 `_CcFileSessionStore`：基于 vault 的文件型会话存储，实现 append（按 uuid 去重）/ load / list / delete / list_subkeys，并对路径做 `_safe_parts` + `resolve()` 防越界校验。 - `_build_options()`：统一构建 `ClaudeAgentOptions`，处理 skills、disallowed_tools（默认禁 `WebSearch`）、`.env` 注入、Claude Code 的 API 凭据解析（`_claude_code_api_env`，多级 base_url/api_key 回退）、`CLAUDE_CONFIG_DIR` 设置、skill 目录软链接（`_ensure_claude_skill_dir`）。 - `_raw_event_to_chunk()` / `_message_content_to_chunks()`：把 Anthropic 流式事件（message_start/delta/stop、content_block_*）与 SDK 消息块（AssistantMessage/UserMessage/ResultMessage/RateLimitEvent）转换为统一 `StreamChunk`；跟踪 block_id/block_type/tool_call_name 做关联；处理尾部 `"success"` 误报异常的吞掉逻辑。 ### 2. 统一流式协议（StreamChunk / ChunkEnum） - **`stream_chunk.py`**：`StreamChunk` 扩展为承载 AS + CC 双后端完整信息的统一结构，新增 `session_id` / `block_id` / `tool_call_id` / `tool_call_name` / `media_type` / `input_tokens` / `output_tokens` 等字段，纯文本流仍保持轻量。 - **`chunk_enum.py`**：补全生命周期标记 `REPLY_START` / `REPLY_END`，并文档化两套后端事件 → ChunkEnum 的映射。 ### 3. Index 模块重构（变化批次化 + dispatch） - 新增 `_change_batch.py`：`coalesce_changes()` 把同路径多次事件折叠为最终状态（结合 path 存在性判定），`bucket_changes()` 按 watchfiles.Change 分桶。 - 新增 `init_changes.py`（`InitChangesStep`）：一次性扫描，对比 file_store / file_catalog 已索引节点计算 added/modified/deleted，写入 `context["changes"]` 后 dispatch。 - 新增 `update_changes.py`：抽象基类 `ChangeApplyStep` 统一 added/modified/deleted 处理与错误收集；`UpdateCatalogStep`（写 file_catalog）、`UpdateIndexStep`（写 file_store，含按后缀解析 chunker）。 - **`watch_changes.py`**：改用 `dispatch_step_specs`（基类提供的 `dispatch_steps()`），每批先 `coalesce_changes` 再 dispatch；默认参数调整（debounce 5000ms / step 1000ms / poll 5000ms）并暴露常量。 - 删除旧步骤：`clear_and_scan` / `foreach_dispatch` / `scan_changes` / `update_catalog`(旧) / `update_index`(旧)；`clear_store.py` 取代 clear_and_scan。 ### 4. Evolve / Dream 模块（拆分为多步 pipeline） - 删除旧的单体 `auto_dream.py` / `dream.py` / `dream.yaml`，新增 `dream/` 子包，按 5 个步骤组织： - **`extract.py`**：扫描当日 day-index + daily 笔记，对比 file_catalog 找出 changed/deleted，调用 LLM 全局抽取 `units`（procedure/personal/wiki 三桶）与 `topics`，路径与桶做清洗/路由。 - **`integrate.py`**：逐个 unit 调用 LLM 写入 digest，结构化输出 `IntegrateOutcome`（CREATE/CORROBORATE/REFINE/CORRECT），失败 unit/路径收集回写。 - **`topics.py`**：写 `daily/<date>/interests.yaml`，结合当天已有 + 近 N 天做去重（`normalize_topic`），可走 LLM 或纯规则去重两条路径。 - **`proactive.py`**：读取当日 `interests.yaml`，作为主动推荐话题的入口。 - **`finish.py`**：把变更路径落盘到 dream file_catalog（checkpoint），渲染最终汇总摘要。 - 新增 `schema.py`（`DreamState` 等跨步骤共享状态与结构化输出模型）与 `utils.py`（状态存取、扫描打包、YAML 读写、结构化回复解析等公共函数）。 - `evolve/__init__.py` 导出全部新 step。 ### 5. auto_memory / auto_resource（适配新 Agent API） - **`auto_memory.py`**：会话路径迁移到 `<session_dir>/dialog/<session_id>.jsonl`；改用 `job_tools`；新增 `source_conversation` frontmatter 反向链接（`_session_link`）；执行后刷新 day 索引（`refresh_day_index`），并对 session_id 做合法性校验。 - **`auto_resource.py`**：资源改用「同名 daily note」方案（`_compute_note_stem` 取文件 stem）；批量处理 `changes: list[dict]`（`_handle_change` 逐项处理，返回逐项结果摘要）；agent 会话 id 用稳定的 `uuid5`；同样刷新 day 索引。 ### 6. BaseStep 基类增强 - 新增 `dispatch_steps` / `dispatch_step_specs` 机制：`_resolve_dispatch_step()` 支持字符串或 dict 形式的 step spec，`dispatch_steps()` 复用当前 context 调用下游 step。 - 新增 `config_value()`：按 key 取 app config，缺失时回退 `ApplicationConfig` 默认值。 - 小幅清理：`language` 初始化、`copy()`、`Ref.__init__` 签名精简。 ### 7. Components 改动 - **`file_store/local_file_store.py`**：持久化改用 zstd 压缩（`.jsonl.zst`，通过新 `utils/jsonl_zst.py`）；upsert 时先删除旧 chunk 的 keyword 文档；embedding 复用改为 `(text, embedding)` 键控，要求文本一致才复用；新增 `_matches_search_filter()` 对 vector/keyword 搜索做 path/path_prefix/metadata 的统一后过滤。 - **`keyword_index/bm25_index.py`**：索引文件名加入组件名 + tokenizer 指纹（sha256 前 12 位），快照/恢复时校验指纹防配置漂移；空索引 dump 时删除文件，加载失败抛错而非静默。 - **`file_chunker/markdown_file_chunker.py`**：弃用 `python-frontmatter`，改用内置 YAML 解析（非法 YAML 不阻断正文索引），并修正因 frontmatter 占用行号导致的 AST 行号偏移（`line_offset`）。 - **`cron_job.py`**：大幅简化（-187 行），由原来「dispatch 外部 job/step + 多种调度模式」改为「在自身 steps 上跑 cron 表达式」；`Application` 启动顺序随之调整为 base > stream > background > cron。 - 其余小调整：service（base/http/mcp）、file_graph、file_catalog、as_llm、as_embedding、tokenizer、prompt_handler、base_component 的签名/接口微调。 ### 8. Application 生命周期 - `_start()` 启动顺序明确为 components → base → stream → background → cron，启动失败会触发 `_close()` 回滚并 re-raise（不再吞异常）。 - 启动时创建 `session_dir` 目录；新增 `update_component()`（按类型/名就地更新已存在组件，不存在则报错）。 ### 9. File IO / 路径安全 - **`_path.py`**：`resolve_path` 增加 vault 越界防护（`is_relative_to` 校验），禁止 `.` / `..` 路径分量，支持 `allow_empty`。 - **`read.py`**：大文件（超过 `MAX_FILE_READ_BYTES`）走按行读取 `read_file_lines_safe`，避免一次性载入内存。 - **`_file_io.py` / `_daily_index.py` / `_path.py`** 等支持函数补齐（如 `refresh_day_index`、`read_file_lines_safe`）。 - **`env_utils.py`**：新增 `parse_env_file()`，`load_env()` 返回加载到的键值、支持 `override`、对无路径调用做幂等缓存。 ### 10. Config - `ApplicationConfig` 新增 `session_dir`（默认 `reme_session`）。 - `config_parser.py`：环境变量展开后做类型转换（`_convert_value`）、dot-notation 与 key=value 参数校验更严格、配置文件路径支持相对 `_CONFIG_DIR` 查找、根非 dict 报错。 - `default.yaml`：作业编排改用 `init_changes_step` + `dispatch_steps`（index/resource/digest 三个 watch loop 与 reindex）；新增 `auto_dream`（4 步）、`proactive` 作业，移除旧 `dream`；file_catalog 增配 `resource` / `digest` / `dream` 实例；LLM 默认值与 Claude Code 凭据配置调整（tool_result_limit 50000、thinking_enable=false 等）。 ### 11. 其它 - 新增 `steps/common/add.py`（`AddStep` 算术 demo）、`channel/__init__.py` 与 common `__init__` 导出整理。 - 新增 4 篇文档：`docs4/auto_dream_logic_and_step_refactor.md`、`docs4/watch_loop_step_refactor_plan.md`、`docs4/todo.md`，以及 `reme_design.md` 更新。 **	2026-06-19 01:35:31 +08:00
Sen Huang	f458566e2c	feat: add cron scheduling support and enhance Claude Code integration (#278) ... * feat: add cron scheduling support and enhance Claude Code integration - Introduce CronStep for periodic job execution with support for cron expressions, daily schedules, and fixed intervals - Add automatic session management to Claude Code agent wrapper with cache-friendly defaults for system prompts and setting sources - Implement fork session support with proper validation - Enhance auto-dream functionality to dispatch per-file jobs instead of direct method calls for better backend agnosticism - Add session ID tracking to auto-resource operations - Remove deprecated download step component - Update auto-dream job naming from auto-dream to auto_dream - Add croniter dependency and update package data to include markdown files * feat: add CronJob component and rename cron step to cron job	2026-06-10 20:40:18 +08:00
jinliyl	c3fb825af0	feat(agent): refactor agent wrapper, add session persistence, auto_resource step, and watch-loop improvements (#277) ... * refactor(agent_wrapper): update agent wrapper implementations and config defaults - Set default timezone to Asia/Shanghai in application config - Add AgentScope imports and configure ReAct, context, and model configs - Simplify __all__ export formatting in agent wrapper init - Remove redundant docstring details from agent wrapper classes - Optimize tool result handling with state assignment simplification - Add permission context and state management for AgentScope backend - Update Claude Code wrapper tool creation and server registration logic - Configure default agent settings including permission mode and retry limits - Remove obsolete comments and streamline code structure * fix(agent): add output schema validation and BaseModel support - Added type assertion to ensure output_schema is a dict in as_agent_wrapper - Imported BaseModel from pydantic in base_agent_wrapper - Modified set_output_schema to accept both dict and BaseModel types - Added automatic conversion of BaseModel to JSON schema - Updated method documentation to reflect new type support * refactor(agent): replace direct agent instantiation with agent wrapper component - Removed manual Agent creation and initialization in llm_demo step - Integrated agent_wrapper component as dependency in base step - Updated llm_demo step to use agent_wrapper.reply method instead of direct agent calls - Modified structured output handling to work with new agent wrapper interface - Simplified agent configuration by using wrapper's built-in functionality - Updated documentation to reflect agent wrapper usage instead of direct as_llm access - Removed redundant imports related to manual agent management * feat(agent): add streaming support and refactor agent wrapper components - Introduce reply_stream method in base agent wrapper with fallback implementation - Add _build_agent helper method to AsAgentWrapper for agent instantiation - Implement structured output generation with proper model assertions - Update StreamLLMDemoStep to use agent_wrapper instead of direct Agent calls - Replace manual streaming logic with execute_stream_task utility function - Change default system prompt to provide detailed responses instead of concise ones - Add colored output support for different chunk types in streaming demos - Refactor test cases to use async task execution with streaming verification * refactor(agent): remove session_id parameter from reply methods - Removed session_id parameter from ASAgentWrapper.reply method signature - Removed session_id parameter from BaseAgentWrapper.reply abstract method - Removed session_id parameter from CCAgentWrapper.reply method signature - Updated reply_stream methods to remove session_id parameter across all wrappers - Modified CCAgentWrapper to use dynamic options assignment instead of hardcoded properties - Set default system_prompt in config instead of hardcoded in code - Increased default max_turns from 10 to 50 in configuration * config: update default configuration and script entry point - Change resource_dir from empty string to 'resource' - Update command line entry point from 'reme4' to 'reme' * feat(agent): add session state persistence and forking support - Implement AsStateHandler for AgentState JSONL serialization - Add session_id parameter to AsAgentWrapper.reply method - Create timestamp-based session file paths with timezone support - Load existing session state from JSONL files when session_id provided - Save updated session state after each agent interaction - Support session forking with UUID generation for new sessions - Add integration tests for session persistence and forking scenarios - Include temporary directory utilities for testing isolated sessions - Ensure parent directories are created for session files automatically * refactor(auto_memory): replace transcript parsing with direct message handling - Remove transcript loading logic and related dependencies - Add session message saving functionality with deduplication - Use agent wrapper instead of direct AgentScope agent instantiation - Simplify timezone handling using shared now utility - Update logging and response metadata structure - Remove unused imports and toolkit management methods - Change session file naming from session_{id}.jsonl to session_agent_{id}.jsonl * refactor(steps): move channel steps from index to channel module - Move ChannelNotifyStep from .index.channel_notify to .channel.channel_notify - Move ClaimChannelStep from .index.claim_channel to .channel.claim_channel - Update __init__.py imports to reflect new module structure - Reorganize steps list in __init__.py with channel section before index - Add proper file prefix handling in daily index processing - Update test imports to use new channel module location * feat(evolve): add auto_resource step for interpreting resource files - Add AutoResourceStep to interpret resource files into daily notes via an agent - Implement resource file parsing with date and filename extraction logic - Add session ID computation using MD5 hash of filename - Create delete and upsert handlers for resource file operations - Add truncation and sanitization functions for tool output in auto_memory - Register auto_resource step with proper parameter validation - Add configuration for resource watch loop with file extension filters - Update default YAML config to include resource watch and digest watch loops - Add shared watch-rule logic for scan_changes and watch_changes steps - Implement foreach_dispatch and log_changes steps for change processing - Rename update_store_index_loop to index_update_loop in configuration - Refactor file chunking interface from parse to chunk method - Remove unused imports and dependencies in auto_dream step - Fix path iteration formatting in daily_index utility function - Add comprehensive integration tests for auto_resource functionality * refactor(auto_resource): format function call with multi-line parameters - Reformatted await _handle_upsert call to use multiple lines for better readability - Removed unused imports from scan_changes.py including BaseFileCatalog and ComponentEnum - Added date parameter to RuntimeContext initialization in test cases - Updated expected file paths in test assertions to include session_agent prefix - Formatted long assertion statements across multiple lines to maintain character limit - Corrected wikilink references from generic names to session_agent prefixed names	2026-06-08 16:11:32 +08:00
jinliyl	8eaa96390a	refactor(file_chunker): replace file parser with file chunker component (#276) ... * refactor(file_chunker): replace file parser with file chunker component - Rename file_parser module to file_chunker across codebase - Update BaseFileParser to BaseFileChunker with corresponding component type - Rename LinkedFileParser to MarkdownFileChunker for markdown-specific chunking - Rename ChunkedFileParser to DefaultFileChunker for default byte-based chunking - Update documentation references from file_parser to file_chunker - Modify dependency injection in BaseStep to use file_chunker instead of file_parser - Update configuration and component registration to use new chunker naming - Rename all related test files and update test assertions accordingly - Add recursive option to scan_store_changes_step in default configuration * feat(database): enhance Neo4j connection with environment variable support - Add support for NEO4J_PASSWORD environment variable as fallback - Make password parameter optional in constructor with validation - Update chromadb dependency from 1.3.5 to 1.5.7 - Configure CORS credentials based on origin settings - Import os module for environment variable access * feat(config): add timezone support and remove unused dialog directory - Added timezone field to application config with IANA timezone support - Removed unused dialog_dir configuration and related directory creation - Replaced date.today() with timezone-aware now() function across daily operations - Created evolve module with timezone-aware datetime functionality - Updated daily_create, daily_list, and daily_reindex steps to use timezone-aware dates * refactor(steps): update file chunker implementation - Replace ChunkedFileParser with DefaultFileChunker in background steps - Add module docstring to evolve steps package - Update return type annotation to reflect new chunker class usage * refactor(components): rename embedding and llm components to as_embedding and as_llm - Rename reme4/components/embedding to reme4/components/as_embedding - Rename reme4/components/llm to reme4/components/as_llm - Update all imports and references from embedding to as_embedding - Update all imports and references from llm to as_llm - Change BaseEmbedding to BaseAsEmbedding and update inheritance - Change BaseLLM to BaseAsLLM and update inheritance - Update component types from LLM/EMBEDDING to AS_LLM/AS_EMBEDDING - Update configuration keys from embedding/llm to as_embedding/as_llm - Update all property references from llm to as_llm in step classes - Update test assertions to use new component enum values * refactor(embedding_store): rename embedding parameter to as_embedding - Updated configuration key from 'embedding' to 'as_embedding' - Renamed class attribute from 'embedding' to 'as_embedding' - Updated method calls to use 'as_embedding' instead of 'embedding' - Changed parameter name in constructor from 'embedding' to 'as_embedding' - Updated documentation to reflect new parameter name - Modified health check to use 'as_embedding' property * feat(agent_wrapper): add unified agent wrapper component with multiple backends - Introduce BaseAgentWrapper abstract base class for agent implementations - Add AsAgentWrapper implementation using AgentScope framework - Add CcAgentWrapper implementation using Claude Code SDK - Register agent_wrapper component type in ComponentEnum - Configure default agent_wrapper settings in default.yaml - Implement tool integration for both AgentScope and Claude Code backends - Support fluent configuration via set_system_prompt() and add_tools() methods * feat(agent-wrapper): add structured output support for agent wrappers - Import SystemMsg in AsAgentWrapper for structured output handling - Add output_schema parameter support in AsAgentWrapper with generate_structured_output - Implement set_output_schema method in BaseAgentWrapper for chaining configuration - Add output schema support in CcAgentWrapper with JSON schema format option - Return structured output when available in CcAgentWrapper response - Refactor kwargs handling to use default values consistently across wrapper classes	2026-06-05 17:27:54 +08:00
Sen Huang	a2d76cc034	refactor(auto_dream): improve recall workflow and documentation (#272) ... * feat(file_store): add concurrency protection to LocalFileStore.dump() * feat(file_catalog): replace file_store with file_catalog in DreamStep * feat: add ChannelSink for Claude Code channel notifications * feat(auto-memory): add transcript_path support and enhance metadata	2026-06-04 19:56:50 +08:00
jinliyl	d8086039dc	refactor(Agentscope2.0): llm & embedding & agent (#271) ... * refactor(embedding): replace embedding model with embedding store architecture - Remove as_token_counter component and its estimated token counter implementation - Replace BaseEmbeddingModel with BaseEmbedding that wraps AgentScope embedding models - Add support for multiple embedding providers (OpenAI, DashScope, Gemini, Ollama) - Introduce BaseEmbeddingStore and LocalEmbeddingStore for caching and persistence - Update component registry to use new embedding and embedding_store types - Modify file stores to use embedding_store instead of embedding_model - Update health check to monitor embedding_store instead of embedding_model - Change default config to use embedding_store with local backend - Add estimate_token_count utility function to utils module * refactor(llm): replace as_llm components with unified llm implementation - Remove deprecated as_llm and as_llm_formatter modules - Add new llm module with BaseLLM and provider-specific implementations - Update component registry to use LLM instead of AS_LLM - Replace all as_llm/as_llm_formatter references with llm in steps - Update configuration schema to use llm instead of as_llm - Rename integration test file from test_as_llm to test_llm - Add proper docstrings to embedding store dimension property - Add pylint disable comment for embedding model call - Remove unused FormatterBase import in base_step - Update token_utils with function docstring * refactor(evolve): replace ReActAgent with Agent and update message handling - Removed FlexReActAgent class and direct ReActAgent imports - Updated Agent instantiation to use new constructor parameters - Changed message content to use TextBlock format instead of plain strings - Modified timestamp access from msg.timestamp to msg.created_at - Updated metadata access pattern for structured outputs - Replaced Msg.from_dict with Msg.model_validate in auto_memory.py - Updated test mocks to patch Agent instead of ReActAgent - Changed message serialization from to_dict to model_dump in tests - Moved component references to base class definition - Updated demo tools to return strings instead of ToolResponse objects * feat(step): migrate to FunctionTool and add streaming support - Replace deprecated ToolResponse with FunctionTool in base_step.py - Remove unused TextBlock import from base_step.py - Update job registration to use new FunctionTool API - Add thinking_budget parameter to llm_demo configuration - Introduce StreamLLMDemoStep with streaming output capability - Add structured output support to LLMDemoStep via generate_structured_output - Implement streaming event handling for text/thinking/tool calls - Add integration tests for embedding functionality - Add integration tests for structured output and streaming features - Update tool usage in demo steps to use new function naming convention * fix(ci): correct package installation path in unittest workflow - Updated pip install command to use proper package path "./reme4[dev,core]" - Fixed dependency installation step in CI workflow configuration * chore(workflow): update python versions in unittest workflow - Remove Python 3.10 from test matrix - Add Python 3.11 to test matrix - Add Python 3.12 to test matrix - Keep Python 3.13 in test matrix - Update matrix configuration for better version coverage * fix(health): handle missing dimensions attribute in embedding status - Wrap dimensions access in try-except to prevent AttributeError - Return None when dimensions attribute is not available - Maintain backward compatibility for components without dimensions test(component): add comprehensive tests for BaseComponent and related classes - Add tests for Dependency class including repr and attribute access - Add tests for bind method with various scenarios and edge cases - Add tests for lifecycle management and async context handling - Add tests for standalone and context-bound dependency resolution - Add tests for ComponentMixin path utilities test(common): update LocalFileStore initialization parameter - Change embedding_model parameter to embedding_store in test setup - Update all affected test files consistently test(registry): add complete test suite for ComponentRegistry - Add tests for register method with explicit names and defaults - Add tests for decorator registration pattern - Add tests for get_all method returning copies - Add tests for unregister and clear operations - Add tests for error handling of invalid registrations test(job): add comprehensive tests for BaseJob and BackgroundJob - Add tests for step resolution and exception handling - Add tests for backoff delay calculation with jitter - Add tests for supervisor loop restart behavior - Add tests for task shutdown and cancellation test(prompt): add complete test suite for PromptHandler - Add tests for prompt loading from dictionaries and files - Add tests for internationalization and language fallback - Add tests for flag filtering and variable substitution - Add tests for format validation and error handling test(runtime): add basic tests for RuntimeContext dictionary access - Add tests for item getting, setting and containment checks - Add tests for missing key error handling * feat(evolve): add permission context and agent state management - Import PermissionContext, PermissionMode and AgentState modules - Add state configuration with bypass permission mode to AutoDream agents - Add state configuration with bypass permission mode to AutoMemory agents - Implement static _to_msg method for message validation and formatting - Refactor message processing to use the new _to_msg method - Ensure proper content structure for text blocks in message conversion * style(tests): update test files with linting rules and code improvements - Add missing pylint disable directives for docstring and attribute warnings - Replace lambda expressions with proper function definitions in test cases - Import Path directly instead of using lambda with __import__ - Simplify assertion checks by using truthiness instead of equality to empty dict - Remove unused imports and reorder imports consistently - Format dictionary literals with proper indentation and line breaks	2026-06-03 11:35:43 +08:00
Sen Huang	16d2d84431	feat(dream): replace digester with abstraction-layer dreamer pipeline (#264) ... * feat(dream): replace digester with abstraction-layer dreamer pipeline Reframe digest as the abstract memory layer (details stay in the daily/ resource material; digest holds principles, patterns, precedents reachable via derived_from provenance edges). Replaces the old digester with a 2-phase ReAct workflow + a daily-tick wrapper: - Phase 1 (Dreamer extract): clusters material into orthogonal memory sub-units; each sub-unit maps 1:1 to a digest node (no inner atom enumeration). Biases toward fewer / richer sub-units. - Phase 2 (Dreamer integrate per sub-unit): cross-bucket recall + exactly one write decision (CREATE / UPDATE / SKIP); UPDATE shapes surfaced explicitly (corroborate / refine / correct). - CronDreamer: scans <daily_dir>/<today>.md + <daily_dir>/<today>/** + <resource_dir>/<today>/** and runs dream_one per file. Write tools are proper subclasses of the canonical file_io WriteStep / EditStep with only path-shape + bucket + E-1 edge-conservation rules layered on top: - DigestWriteStep(WriteStep): path = <digest_dir>/<bucket>/<slug>.md, must-not-exist, schema mirrors `write` (path / name / description / content) so frontmatter lands automatically. - DigestEditStep(EditStep): body-only find-and-replace + must-exist + E-1 conservation preflight (refuses if any outbound wikilink would be dropped). Configuration: - Bucket vocabulary structured in code (tuple[{name, description}]); prompt renders the heuristic block at runtime via {buckets}. - digest_dir / daily_dir / resource_dir come from app config (not tool params); prompts use {digest_dir} placeholder. - BaseStep walks class MRO when loading prompts, so subclasses inherit parent yaml without duplication. Tooling: agentscope register_tool_function schemas now wrap in the proper {"type":"function","function":{...}} envelope. OpenAIAsLLM routes base_url through client_kwargs so non-default endpoints work. Smoke: tests4/smoke/{_dreamer_fixture.py,test_dreamer_inproc.py, test_dreamer_cli.sh} drive the end-to-end pipeline. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(dreamer): split long description string across multiple lines * refactor(dream): remove hardcoded DEFAULT_DIGEST_DIR and use app_config * docs(auto-cognition): add comprehensive design document for auto-cognition system * refactor(steps): remove deprecated digest edit/write steps * refactor(config): remove redundant LLM formatter backend configuration * refactor(dreamer): improve code formatting and line breaks * feat(auto-dream): implement three-bucket classification system for knowledge organization * feat: rename dream_today step to auto-dream and refactor extraction logic	2026-06-01 19:09:59 +08:00
jinliyl	aa87f4fdea	feat(base_step): set default language from app context when not provided (#269) ... - Initialize language attribute with app context language if not explicitly set - Add conditional logic to check for existing language value before assignment - Ensure proper fallback behavior when language parameter is empty or None	2026-06-01 17:14:20 +08:00
jinliyl	041f957a7f	refactor(components) components and file I/O, fix method calls and validation (#268) ... * refactor(components): extract shared component state into mixin - Introduce ComponentMixin class with shared state for components and steps - Move identity, config, and vault path functionality to ComponentMixin - Update BaseComponent to inherit from ComponentMixin - Update BaseStep to inherit from ComponentMixin - Consolidate vault path helper methods in ComponentMixin - Remove duplicate vault path implementations from BaseComponent and BaseStep - Add ComponentMixin to components module exports * refactor(file_io): implement path locks cache eviction mechanism - Add _PATH_LOCKS_MAX constant set to 1024 for cache size limit - Implement cache eviction logic when locks exceed maximum capacity - Remove half of unlocked entries when cache limit is reached - Use list comprehension to identify unlocked locks for removal - Maintain existing path normalization and locking behavior fix(edit): correct method call from public to private fail method - Change self.fail to self._fail for internal error handling - Maintain consistent private method usage within class fix(mcp_client): change pop to get for optional command and args - Replace kwargs.pop with kwargs.get to avoid removing keys - Preserve original kwargs dictionary contents - Maintain default empty string and list values feat(reme): add client backend validation with error raising - Check if client_cls is None before instantiation - Raise ValueError with descriptive message for unknown backends - Provide clear error feedback for invalid backend configurations * fix(components): move directory creation to start method - Moved component_metadata_path.mkdir call from __init__ to _start in base_keyword_index - Moved component_metadata_path.mkdir call from __init__ to _start in local_file_graph - Moved component_metadata_path.mkdir call from __init__ to _start in local_file_store - Ensures directory creation happens after component initialization - Prevents potential issues with path creation during object construction * fix(steps): replace assertions with runtime errors for app_context validation - Replace assert statements with explicit RuntimeError exceptions when app_context is None - Add descriptive error messages for better debugging when resolving components - Replace assert in resolve_component method with proper exception handling - Replace assert in get_file_parser method with proper exception handling - Maintain same functionality while improving error reporting clarity * refactor(file_io): split file IO utilities into modular components - Move daily note helpers to separate _daily_index module - Extract path validation and resolution to new _path module - Remove unused code and imports from _file_io module - Update import statements across affected modules - Introduce WikilinkHandler utility for link parsing - Replace regex-based link extraction with WikilinkHandler - Add integration JSONL files to gitignore - Consolidate file locking mechanism in _file_io module * style(formatter): fix spacing issues in file IO and chunked file parser - Fixed whitespace around colon in slice notation in file_io.py - Corrected spacing around colon in slice notation in chunked_file_parser.py - Applied consistent formatting for array slicing operations - Improved code readability by standardizing space placement in ranges * refactor(steps): replace property-based component resolution with Ref descriptor - Introduce Ref descriptor class for lazy component dependency resolution - Replace _resolve method and individual properties with Ref descriptors - Add as_llm, as_llm_formatter, as_token_counter, file_store, and embedding Ref attributes - Remove legacy property methods and resolve logic from BaseStep - Add cache clearing mechanism for Ref values during step calls - Update UpdateCatalogStep to use Ref instead of property-based resolution	2026-06-01 11:35:19 +08:00
jinliyl	c4ca617992	refactor(evolve): consolidate auto memory planner and writer into single step (#267) ... * refactor(evolve): consolidate auto memory planner and writer into single step - Removed separate AutoMemoryPlannerStep and AutoMemoryWriterStep classes - Combined functionality into new AutoMemoryStep class in auto_memory.py - Migrated prompt templates from separate YAML files to unified auto_memory.yaml - Updated module imports to reference new consolidated step - Simplified memory recording process using single ReAct agent instead of two-stage planning/writing - Maintained same input/output contract with messages, session_id, and memory_hint parameters - Preserved all original functionality for creating/updating daily notes with conversation facts * fix(daily): update empty session_id handling to create day-level file - Changed test to verify empty session_id creates day-level file daily/<date>.md - Updated assertion to check response success instead of rejection - Modified metadata verification to include path, session_id and created status - Added file existence check for the generated daily markdown file - Updated test name and print statement to reflect new behavior - Fixed test registration to use updated function name	2026-05-29 18:02:10 +08:00
jinliyl	9ee2f0f7ab	refactor(daily): replace slug with session_id for daily note identification (#266) ... - Rename slug parameter to session_id across daily note operations - Update validation function from validate_slug to validate_session_id - Change data structure keys from slug to session_id in note objects - Modify file paths to use session_id instead of slug in daily folder - Update documentation and comments to reflect session_id terminology - Adjust test cases to use session_id parameter instead of slug - Change default frontmatter to include empty description field - Update configuration files to use session_id parameter name - Modify scan_notes function to return session_id instead of slug	2026-05-29 16:00:51 +08:00
jinliyl	ef22bfb071	refactor(evolve): replace ReActAgent with FlexReActAgent to allow structured output (#265) ... - Create FlexReActAgent subclass that overrides _reasoning method to handle tool_choice parameter - Modify auto_memory_planner to use FlexReActAgent instead of ReActAgent - Update base_step.py to accept additional kwargs in add_as_tool method - Change run_job function to merge kwargs properly when calling jobs - Remove redundant imports and constants from file_io.py - Simplify _render_notes_block and rename _replace_or_append_notes to _rebuild_body - Update method calls to use new function names in file_io operations	2026-05-29 15:19:56 +08:00
jinliyl	3cb2579ff7	refactor(steps): update auto-memory (#263) ... * refactor(steps): update naming conventions in components and configuration Updated naming conventions across multiple files, changing colon-separated names to underscore-separated format, and added new step definitions along with documentation updates. Key changes: - Replaced `Synchronizer` with `AutoMemory` as the counterpart component for cold-write operations - Updated naming conventions in all related configuration files (e.g., `frontmatter:read` → `frontmatter_read`) - Added new step definitions such as `submit_slug_updates` and `auto_memory` - Updated relevant documentation - Modified log output format for improved readability * refactor(evolve): Refactor the auto-memory module and update related configurations - Remove the old slug update commit step file - Add new auto-memory planner and writer steps - Update __init__.py to export the new step classes - Modify the auto_memory configuration structure in default.yaml - Update the slug field description for clearer explanation of its purpose * up * up * refactor(tests): Move unit test directory from `tests4/unittest` to `tests4/unit` Additionally, the assertion logic in test files has been updated: direct comparisons of `payload["notes"]` have been replaced with checks verifying the presence of paths and metadata within the response content. Furthermore, some test expectations have been simplified—for example, using `count` instead of asserting against specific note lists. Specific changes include: - Updating workflow configurations to align with the new test directory structure - Modifying assertions across multiple test methods to make them more flexible and maintainable - Cleaning up and optimizing parts of the test code structure This is a comprehensive test refactoring effort aimed at improving test readability and robustness. * Refactor(steps): Update memory writing logic and optimize JSON schema structure Improved the write strategy description in `auto_memory_writer.yaml` to emphasize using `edit` over `write`. Adjusted the `json_schema` structure in `base_step.py` to support the new function definition format. Also corrected grammatical issues in the related documentation. * Fix: Improve frontend data parsing error handling and update test files Added capture and handling logic for YAML parsing exceptions, providing more detailed error messages when frontend data format issues occur. Also corrected the description text in a test file.	2026-05-29 12:07:44 +08:00
imrewce	2ed2e89e24	port orthogonal steps (#262) ... * feat(file_io): port orthogonal crud_steps features onto upstream restructure * refactor(file_io): expose with_neighbors/max_neighbors_per_direction/max_bytes as step kwargs (not LLM params) Match search_step's convention: tuning knobs that are config-like (not part of the LLM-facing schema) live in the yaml steps: block and are read via self.kwargs.get(...) — not exposed under parameters.properties. Also simplify the write step metadata field description.	2026-05-28 18:10:04 +08:00
Sen Huang	8c48798164	feat(jobs): add digester step for knowledge distillation from daily notes (#261) ... * feat(jobs): add digester step for knowledge distillation from daily notes * feat: add initial implementation * refactor(jobs): remove unnecessary blank lines in digester and synchronizer * feat: add initial implementation	2026-05-28 15:17:23 +08:00
jinliyl	a4efc0f776	refactor(reme4): restructure steps packages (#258) ... * fix(bm25_index): 修正BM25索引计算中的文档长度归一化问题 修复了在计算BM25相似度时对文档长度进行不正确归一化的bug，确保所有查询都能得到准确的相关性评分。 * up * up * up * up * up * up * up * up * up * up * up * up * up * up * up * up * up * refactor(steps): Rename and adjust indexing step logic - Rename `scan_changes.py` and `reindex.py` to `clear_and_scan.py` - Update implementation details of `ScanChangesStep` and `ClearAndScanStep` - Modify the scheduling mechanism in `WatchChangesStep` - Adjust step registration and parameter configuration in config files - Update related tests to align with the new interface changes * up * feat(daily): replace daily CRUD operations with slug provisioning approach * refactor(tests): migrate CRUD step tests from HTTP server to direct LocalFileStore * up * up * up * up --------- Co-authored-by: huangsen <huangsen.huang@alibaba-inc.com>	2026-05-28 14:30:30 +08:00
Sen Huang	83bfddb4a4	refactor(config): streamline job descriptions and parameter docs (#257) ... * refactor(config): streamline job descriptions and parameter docs - Simplify descriptions for search, traverse, list, read, stat, frontmatter:read, write, edit, append, frontmatter:update, frontmatter:delete, move, delete, upload, upload_resource, and download jobs - Shorten parameter descriptions to be more concise - Maintain essential information while reducing verbosity refactor(steps): rename daily steps and consolidate functionality - Rename daily_resolve_step to daily_read_step - Rename daily_create_step to daily_write_step - Update __init__.py imports to reflect new step names - Consolidate daily operations documentation refactor(daily): extract helper functions and improve structure - Rename _day_index.py to _daily_io.py - Extract validate_slug function for Windows-safe filename validation - Move scan_notes function to public interface - Add comprehensive docstrings explaining slug validation and day-index rebuild concerns feat(daily): decouple list operation from index refresh - Remove automatic day index refresh from daily_list_step - Change daily_list_step to pure read operation with no side effects - Sort notes by slug for stable output - Update documentation to clarify read/write separation refactor(daily): remove deprecated create step - Remove unused daily/create.py module - Simplify daily operations to focus on CRUD patterns * refactor(daily): replace module imports with explicit step class imports	2026-05-25 19:20:50 +08:00
Sen Huang	bb354cc580	refactor(steps): reorganize step modules and remove demo steps (#255) ... * feat(config): add comprehensive job definitions for vault operations - Add utility jobs like version, search, traverse, list, read, stat - Include file operations like move, delete, upload, download - Add daily workspace management jobs: daily_list, daily_resolve, daily_reindex - Update descriptions to reflect vault-based operations instead of working_dir - Add proper section headers and documentation for each job category refactor(steps): reorganize step modules and remove demo steps - Move steps into categorized packages: common, crud, frontmatter, daily, jobs - Remove demo steps (DemoEchoStep1, DemoEchoStep2, StreamDemoStep1, StreamDemoStep2) - Add new steps: InitStep for vault initialization, TraverseStep for graph traversal - Update __init__.py to auto-import all step modules - Organize imports by functionality (common, CRUD operations, frontmatter, daily) feat(vault): implement vault-centric file operations and configuration - Change default config to use vault_dir instead of working_dir - Add environment variable support for embedding configuration - Implement file watcher with lite backend for daily/digest directories - Update search step to use 'name' instead of 'title' from frontmatter - Create ResourceEntry schema for tracking uploaded assets docs(steps): add comprehensive documentation for all step categories - Document file-I/O split by blast radius (crud vs frontmatter packages) - Add detailed descriptions for each step category and functionality - Explain the purpose and usage patterns for different types of file operations - Provide clear parameter documentation for all new job configurations * fix(config): correct vault directory path and remove unused job configurations - Fix vault_dir from 'vaultd' to 'vault' in default configuration - Remove deprecated traverse and list job configurations - Remove unused tag tooling configurations - Remove background watch_file job configuration refactor(steps): remove unused jobs module import - Comment out jobs module import in steps/__init__.py - This removes unused synchronizer and digester step registrations refactor(tests): update import path and add pylint directive - Update ResourceEntry import from reme4.schema to reme4.schema.resource_meta - Add pylint disable directive for unused argument in test datetime mocks * efactor(steps): remove unused modules from __all__ - Remove "background" module from __all__ list - Remove "jobs" module from __all__ list - These modules were no longer being used in the steps package * feat(config): update vault directory structure and remove file watcher - Change vault_dir reference from ./vault to ./vault in CLI example - Add daily_dir, digest_dir, and resource_dir configuration options - Remove file_watcher component configuration as it's no longer needed - Update comment to reflect correct module name (reme4vault) refactor(steps): add background step and remove deprecated init step - Import and register background step module - Remove deprecated InitStep from common steps - Update __all__ export list to include background step refactor(reindex): improve reindex step to scan vault directly - Update docstring to reflect vault scanning instead of watcher sync - Replace file watcher stop/start logic with direct vault path walking - Add support for suffix filtering during reindex operation - Use index_changes job to process found files refactor(wikilink_utils): enhance inbound source lookup with link scope - Import LinkScopeEnum for proper type handling - Update get_inlinks call to use ALL scope for virtual targets - Improve documentation for reverse-index lookup behavior test(refactor): clean up test suite removing deprecated functionality - Remove test_init_job and test_demo_job unit tests - Update help job assertion to check for literal command format - Change test directory from .reme to vault in CRUD tests - Remove init and demo job calls from integration test BREAKING CHANGE: Removes file_watcher component and init step * style(steps): fix import formatting in __init__.py Add proper spacing in the background module import statement to maintain consistent code style and readability. * refactor(config): change default vault directory from vault to .reme Default dev config now points vault_dir at ./.reme so `python -m reme4 start` can be run from the repo root and exercise the full atomic-tool surface against the seeded test data. BREAKING CHANGE: The default vault directory has been changed from 'vault' to '.reme' in the configuration. * docs(reme4_report): fix markdown formatting and remove extra content * refactor(file_parser): delegate wikilink extraction to WikilinkHandler * fix(search): handle empty query case gracefully - Replace assertion with conditional check for empty query - Set response success to false when query is empty - Return error message instead of throwing assertion error - Maintain existing validation for other parameters	2026-05-25 17:52:51 +08:00
Sen Huang	7d0bec60be	feat: rename working_dir to vault_dir and update documentation (#254) ... * feat: rename working_dir to vault_dir and update documentation - Rename working_dir to vault_dir across the application - Update documentation to reflect vault_dir instead of working_dir - Change FileFrontMatter title field to name field - Update .gitignore to include vault directory - Modify file path descriptions to reference vault instead of working_dir - Update related configuration and property names accordingly * refactor(steps): rename working_path to vault_path in CRUD operations - Rename parameter from `working_path` to `vault_path` in `resolve_path` function - Update all usages in append, edit, read, and write steps to use `self.vault_path` - Update documentation comments to reflect the new parameter name - Update docstring in read.py to mention `vault_dir` instead of `vault` test(chunked_file_parser): update frontmatter field from title to name - Change frontmatter field from `title` to `name` in test cases - Update comment in background steps test to reference `vault_path` instead of `working_path` * refactor(schema): remove unused ResourceEntry import * feat(file_graph): add link scope filtering to get_inlinks/get_outlinks * feat(file-store): add scope parameter to link methods	2026-05-22 18:10:29 +08:00
jinliyl	24cff10d46	feat(file_store): add FAISS-backed local file store implementation (#253) ... * feat(file_store): add FAISS-backed local file store implementation - Introduce FaissLocalFileStore class with vector search capabilities using FAISS IndexFlatIP - Implement FAISS index persistence with binary format and JSON id-map sidecar - Add automatic index rebuilding when sidecar files are missing or corrupted - Support tombstone mechanism for efficient deletion and compaction - Register 'faiss' component type in the registry system - Add faiss-cpu dependency requirement to pyproject.toml - Update configuration schema to use simplified parameter structure - Enhance search step to support parameter override from runtime context - Add comprehensive unit tests for FAISS store functionality - Implement fallback to parent methods for basic CRUD operations * refactor(search): simplify parameter retrieval logic - Removed _param method that checked context and kwargs - Directly use self.kwargs.get for all parameter retrievals - Maintained same default values for vector_weight, candidate_multiplier, expand_links, and max_links_per_direction - Reduced code complexity by eliminating redundant context checking logic	2026-05-22 16:14:38 +08:00
jinliyl	ee94d3ec8b	docs(reme4): update report with detailed architecture sections (#252) ... - Add comprehensive Markdown kernel section covering Obsidian compatibility - Include detailed explanation of YAML front matter and wikilink formats - Document smart slicing mechanism using Markdown AST instead of fixed tokens - Explain graph indexing with bidirectional links and multiple backends - Restructure sections with proper numbering from 4 to 7 - Move Markdown kernel section to appear before self-evolution features - Add detailed explanations of auto-memory, auto-dream, and auto-link processes - Document three-way hybrid search with RRF fusion and progressive expansion - Include engineering value explanations for keyword indexing in Chinese context	2026-05-22 14:58:37 +08:00
jinliyl	71e42dbad0	refactor(reme4): replace file_watcher component with background steps pipeline (#251) ... * up * up * up * up * up * up * up * up * up * up * up * feat(config): add daily_dir configuration and background job logging - Added daily_dir setting with default value 'memory' to config - Implemented logging for background job startup events - Enhanced component start logic to handle background backend type - Updated default YAML configuration structure * refactor(file_parser): replace _get_relative_path with to_vault_relative method - Remove redundant working_dir property from base file parser - Add to_vault_relative method to base component for path resolution - Update bare_file_parser to use new to_vault_relative method - Update default_file_parser to use new to_vault_relative method - Update linked_file_parser to use new to_vault_relative method - Make working_path absolute in base_component and steps - Simplify index_changes step by removing redundant base variable - Consolidate path relative logic in single shared method * docs(reme4): update report with detailed architecture sections - Add comprehensive Markdown kernel section covering Obsidian compatibility - Include detailed explanation of YAML front matter and wikilink formats - Document smart slicing mechanism using Markdown AST instead of fixed tokens - Explain graph indexing with bidirectional links and multiple backends - Restructure sections with proper numbering from 4 to 7 - Move Markdown kernel section to appear before self-evolution features - Add detailed explanations of auto-memory, auto-dream, and auto-link processes - Document three-way hybrid search with RRF fusion and progressive expansion - Include engineering value explanations for keyword indexing in Chinese context	2026-05-22 10:26:36 +08:00
imrewce	3285934f34	create/ append/ edit steps (#249) ... * feat(core): adding create append edit steps for md crud * fix(core): changing frontmatter args scope for create step * fix(core): chang write frontmatter schema; fix edit logic; fix passing name payload to http client * refractor(steps): changing step compatibility for non-md files	2026-05-21 15:57:37 +08:00
imrewce	e8592fc930	fix: recent unittest inconsistency (#248) ... * feat: implementation of the read step for reme (markdown) * fix(step): markdown read step fixing pr comments * fix(step): more fixes for pr comments * further fix for better review adaptation * accept absolute path * fixing base job exception * fix: fix test inconsistency	2026-05-20 09:58:15 +08:00
imrewce	bee2648ad1	feat: implementation of the read step for reme (markdown) (#245) ... * feat: implementation of the read step for reme (markdown) * fix(step): markdown read step fixing pr comments * fix(step): more fixes for pr comments * further fix for better review adaptation * accept absolute path * fixing base job exception	2026-05-19 16:19:13 +08:00
jinliyl	68bd95b494	refactor(steps): Add job management methods and support registering t… (#242) ... * refactor(steps): Add job management methods and support registering them as tools Added methods to the `BaseStep` class for retrieving, running, and registering jobs as tools, enhancing the functionality of the step class. * fix doc	2026-05-18 15:00:02 +08:00
jinliyl	e411eeb4c0	dev/reme4 init merge (#236)	2026-05-17 14:14:43 +08:00