19 commits

Author	SHA1	Message	Date
jinliyl	83831ec90c	feat(core): enhance reme4 (#281) ... ### 1. Agent Wrapper（统一 Agent 后端抽象） - **`base_agent_wrapper.py`**：`reply()` 返回值从 `tuple[str, Any]` 改为 `dict`（含 `session_id` / `last_message` / `result` / 可选 `structured_output`）；`reply_stream()` 改为产出统一的 `StreamChunk`。废弃 `add_tools()`，改为 `add_job_tools(names: list[str])`（按名解析 BaseJob）与 `add_skills()`；新增 `_resolve_job_tools()`、`_merged_kwargs()`、`_chunk()` 辅助方法及 `project_path` / `project_skills_root` 属性。 - **`as_agent_wrapper.py`（AgentScope 后端）**： - 会话持久化重写：`session_path` 落地到 `<vault>/<session_dir>/agentscope/`，`_load_state` 支持 `resume` / `session_id` / `fork_session`，并做 UUID 校验（`_validate_session_id`）；`_cleanup_expired_sessions` 按天数清理过期会话。 - 新增内置工具集（`BypassAnalysisBash` + Edit/Glob/Grep/Read/Write），`BypassAnalysisBash` 绕过 AgentScope 自带 Bash 静态分析以让 permission_mode 生效；`_resolve_skills()` 把配置的 skill 暴露给后端，`_load_tool_env()` 注入项目 `.env`。 - `_event_to_chunk()` 把 20+ 种 AgentScope 事件（Reply/Text/Thinking/Data/ToolCall/ToolResult/ModelCall/ExceedMaxIters）归一化为 `StreamChunk`。 - **`cc_agent_wrapper.py`（Claude Code SDK 后端，+551 行）**： - 新增 `_CcFileSessionStore`：基于 vault 的文件型会话存储，实现 append（按 uuid 去重）/ load / list / delete / list_subkeys，并对路径做 `_safe_parts` + `resolve()` 防越界校验。 - `_build_options()`：统一构建 `ClaudeAgentOptions`，处理 skills、disallowed_tools（默认禁 `WebSearch`）、`.env` 注入、Claude Code 的 API 凭据解析（`_claude_code_api_env`，多级 base_url/api_key 回退）、`CLAUDE_CONFIG_DIR` 设置、skill 目录软链接（`_ensure_claude_skill_dir`）。 - `_raw_event_to_chunk()` / `_message_content_to_chunks()`：把 Anthropic 流式事件（message_start/delta/stop、content_block_*）与 SDK 消息块（AssistantMessage/UserMessage/ResultMessage/RateLimitEvent）转换为统一 `StreamChunk`；跟踪 block_id/block_type/tool_call_name 做关联；处理尾部 `"success"` 误报异常的吞掉逻辑。 ### 2. 统一流式协议（StreamChunk / ChunkEnum） - **`stream_chunk.py`**：`StreamChunk` 扩展为承载 AS + CC 双后端完整信息的统一结构，新增 `session_id` / `block_id` / `tool_call_id` / `tool_call_name` / `media_type` / `input_tokens` / `output_tokens` 等字段，纯文本流仍保持轻量。 - **`chunk_enum.py`**：补全生命周期标记 `REPLY_START` / `REPLY_END`，并文档化两套后端事件 → ChunkEnum 的映射。 ### 3. Index 模块重构（变化批次化 + dispatch） - 新增 `_change_batch.py`：`coalesce_changes()` 把同路径多次事件折叠为最终状态（结合 path 存在性判定），`bucket_changes()` 按 watchfiles.Change 分桶。 - 新增 `init_changes.py`（`InitChangesStep`）：一次性扫描，对比 file_store / file_catalog 已索引节点计算 added/modified/deleted，写入 `context["changes"]` 后 dispatch。 - 新增 `update_changes.py`：抽象基类 `ChangeApplyStep` 统一 added/modified/deleted 处理与错误收集；`UpdateCatalogStep`（写 file_catalog）、`UpdateIndexStep`（写 file_store，含按后缀解析 chunker）。 - **`watch_changes.py`**：改用 `dispatch_step_specs`（基类提供的 `dispatch_steps()`），每批先 `coalesce_changes` 再 dispatch；默认参数调整（debounce 5000ms / step 1000ms / poll 5000ms）并暴露常量。 - 删除旧步骤：`clear_and_scan` / `foreach_dispatch` / `scan_changes` / `update_catalog`(旧) / `update_index`(旧)；`clear_store.py` 取代 clear_and_scan。 ### 4. Evolve / Dream 模块（拆分为多步 pipeline） - 删除旧的单体 `auto_dream.py` / `dream.py` / `dream.yaml`，新增 `dream/` 子包，按 5 个步骤组织： - **`extract.py`**：扫描当日 day-index + daily 笔记，对比 file_catalog 找出 changed/deleted，调用 LLM 全局抽取 `units`（procedure/personal/wiki 三桶）与 `topics`，路径与桶做清洗/路由。 - **`integrate.py`**：逐个 unit 调用 LLM 写入 digest，结构化输出 `IntegrateOutcome`（CREATE/CORROBORATE/REFINE/CORRECT），失败 unit/路径收集回写。 - **`topics.py`**：写 `daily/<date>/interests.yaml`，结合当天已有 + 近 N 天做去重（`normalize_topic`），可走 LLM 或纯规则去重两条路径。 - **`proactive.py`**：读取当日 `interests.yaml`，作为主动推荐话题的入口。 - **`finish.py`**：把变更路径落盘到 dream file_catalog（checkpoint），渲染最终汇总摘要。 - 新增 `schema.py`（`DreamState` 等跨步骤共享状态与结构化输出模型）与 `utils.py`（状态存取、扫描打包、YAML 读写、结构化回复解析等公共函数）。 - `evolve/__init__.py` 导出全部新 step。 ### 5. auto_memory / auto_resource（适配新 Agent API） - **`auto_memory.py`**：会话路径迁移到 `<session_dir>/dialog/<session_id>.jsonl`；改用 `job_tools`；新增 `source_conversation` frontmatter 反向链接（`_session_link`）；执行后刷新 day 索引（`refresh_day_index`），并对 session_id 做合法性校验。 - **`auto_resource.py`**：资源改用「同名 daily note」方案（`_compute_note_stem` 取文件 stem）；批量处理 `changes: list[dict]`（`_handle_change` 逐项处理，返回逐项结果摘要）；agent 会话 id 用稳定的 `uuid5`；同样刷新 day 索引。 ### 6. BaseStep 基类增强 - 新增 `dispatch_steps` / `dispatch_step_specs` 机制：`_resolve_dispatch_step()` 支持字符串或 dict 形式的 step spec，`dispatch_steps()` 复用当前 context 调用下游 step。 - 新增 `config_value()`：按 key 取 app config，缺失时回退 `ApplicationConfig` 默认值。 - 小幅清理：`language` 初始化、`copy()`、`Ref.__init__` 签名精简。 ### 7. Components 改动 - **`file_store/local_file_store.py`**：持久化改用 zstd 压缩（`.jsonl.zst`，通过新 `utils/jsonl_zst.py`）；upsert 时先删除旧 chunk 的 keyword 文档；embedding 复用改为 `(text, embedding)` 键控，要求文本一致才复用；新增 `_matches_search_filter()` 对 vector/keyword 搜索做 path/path_prefix/metadata 的统一后过滤。 - **`keyword_index/bm25_index.py`**：索引文件名加入组件名 + tokenizer 指纹（sha256 前 12 位），快照/恢复时校验指纹防配置漂移；空索引 dump 时删除文件，加载失败抛错而非静默。 - **`file_chunker/markdown_file_chunker.py`**：弃用 `python-frontmatter`，改用内置 YAML 解析（非法 YAML 不阻断正文索引），并修正因 frontmatter 占用行号导致的 AST 行号偏移（`line_offset`）。 - **`cron_job.py`**：大幅简化（-187 行），由原来「dispatch 外部 job/step + 多种调度模式」改为「在自身 steps 上跑 cron 表达式」；`Application` 启动顺序随之调整为 base > stream > background > cron。 - 其余小调整：service（base/http/mcp）、file_graph、file_catalog、as_llm、as_embedding、tokenizer、prompt_handler、base_component 的签名/接口微调。 ### 8. Application 生命周期 - `_start()` 启动顺序明确为 components → base → stream → background → cron，启动失败会触发 `_close()` 回滚并 re-raise（不再吞异常）。 - 启动时创建 `session_dir` 目录；新增 `update_component()`（按类型/名就地更新已存在组件，不存在则报错）。 ### 9. File IO / 路径安全 - **`_path.py`**：`resolve_path` 增加 vault 越界防护（`is_relative_to` 校验），禁止 `.` / `..` 路径分量，支持 `allow_empty`。 - **`read.py`**：大文件（超过 `MAX_FILE_READ_BYTES`）走按行读取 `read_file_lines_safe`，避免一次性载入内存。 - **`_file_io.py` / `_daily_index.py` / `_path.py`** 等支持函数补齐（如 `refresh_day_index`、`read_file_lines_safe`）。 - **`env_utils.py`**：新增 `parse_env_file()`，`load_env()` 返回加载到的键值、支持 `override`、对无路径调用做幂等缓存。 ### 10. Config - `ApplicationConfig` 新增 `session_dir`（默认 `reme_session`）。 - `config_parser.py`：环境变量展开后做类型转换（`_convert_value`）、dot-notation 与 key=value 参数校验更严格、配置文件路径支持相对 `_CONFIG_DIR` 查找、根非 dict 报错。 - `default.yaml`：作业编排改用 `init_changes_step` + `dispatch_steps`（index/resource/digest 三个 watch loop 与 reindex）；新增 `auto_dream`（4 步）、`proactive` 作业，移除旧 `dream`；file_catalog 增配 `resource` / `digest` / `dream` 实例；LLM 默认值与 Claude Code 凭据配置调整（tool_result_limit 50000、thinking_enable=false 等）。 ### 11. 其它 - 新增 `steps/common/add.py`（`AddStep` 算术 demo）、`channel/__init__.py` 与 common `__init__` 导出整理。 - 新增 4 篇文档：`docs4/auto_dream_logic_and_step_refactor.md`、`docs4/watch_loop_step_refactor_plan.md`、`docs4/todo.md`，以及 `reme_design.md` 更新。 **	2026-06-19 01:35:31 +08:00
jinliyl	c3fb825af0	feat(agent): refactor agent wrapper, add session persistence, auto_resource step, and watch-loop improvements (#277) ... * refactor(agent_wrapper): update agent wrapper implementations and config defaults - Set default timezone to Asia/Shanghai in application config - Add AgentScope imports and configure ReAct, context, and model configs - Simplify __all__ export formatting in agent wrapper init - Remove redundant docstring details from agent wrapper classes - Optimize tool result handling with state assignment simplification - Add permission context and state management for AgentScope backend - Update Claude Code wrapper tool creation and server registration logic - Configure default agent settings including permission mode and retry limits - Remove obsolete comments and streamline code structure * fix(agent): add output schema validation and BaseModel support - Added type assertion to ensure output_schema is a dict in as_agent_wrapper - Imported BaseModel from pydantic in base_agent_wrapper - Modified set_output_schema to accept both dict and BaseModel types - Added automatic conversion of BaseModel to JSON schema - Updated method documentation to reflect new type support * refactor(agent): replace direct agent instantiation with agent wrapper component - Removed manual Agent creation and initialization in llm_demo step - Integrated agent_wrapper component as dependency in base step - Updated llm_demo step to use agent_wrapper.reply method instead of direct agent calls - Modified structured output handling to work with new agent wrapper interface - Simplified agent configuration by using wrapper's built-in functionality - Updated documentation to reflect agent wrapper usage instead of direct as_llm access - Removed redundant imports related to manual agent management * feat(agent): add streaming support and refactor agent wrapper components - Introduce reply_stream method in base agent wrapper with fallback implementation - Add _build_agent helper method to AsAgentWrapper for agent instantiation - Implement structured output generation with proper model assertions - Update StreamLLMDemoStep to use agent_wrapper instead of direct Agent calls - Replace manual streaming logic with execute_stream_task utility function - Change default system prompt to provide detailed responses instead of concise ones - Add colored output support for different chunk types in streaming demos - Refactor test cases to use async task execution with streaming verification * refactor(agent): remove session_id parameter from reply methods - Removed session_id parameter from ASAgentWrapper.reply method signature - Removed session_id parameter from BaseAgentWrapper.reply abstract method - Removed session_id parameter from CCAgentWrapper.reply method signature - Updated reply_stream methods to remove session_id parameter across all wrappers - Modified CCAgentWrapper to use dynamic options assignment instead of hardcoded properties - Set default system_prompt in config instead of hardcoded in code - Increased default max_turns from 10 to 50 in configuration * config: update default configuration and script entry point - Change resource_dir from empty string to 'resource' - Update command line entry point from 'reme4' to 'reme' * feat(agent): add session state persistence and forking support - Implement AsStateHandler for AgentState JSONL serialization - Add session_id parameter to AsAgentWrapper.reply method - Create timestamp-based session file paths with timezone support - Load existing session state from JSONL files when session_id provided - Save updated session state after each agent interaction - Support session forking with UUID generation for new sessions - Add integration tests for session persistence and forking scenarios - Include temporary directory utilities for testing isolated sessions - Ensure parent directories are created for session files automatically * refactor(auto_memory): replace transcript parsing with direct message handling - Remove transcript loading logic and related dependencies - Add session message saving functionality with deduplication - Use agent wrapper instead of direct AgentScope agent instantiation - Simplify timezone handling using shared now utility - Update logging and response metadata structure - Remove unused imports and toolkit management methods - Change session file naming from session_{id}.jsonl to session_agent_{id}.jsonl * refactor(steps): move channel steps from index to channel module - Move ChannelNotifyStep from .index.channel_notify to .channel.channel_notify - Move ClaimChannelStep from .index.claim_channel to .channel.claim_channel - Update __init__.py imports to reflect new module structure - Reorganize steps list in __init__.py with channel section before index - Add proper file prefix handling in daily index processing - Update test imports to use new channel module location * feat(evolve): add auto_resource step for interpreting resource files - Add AutoResourceStep to interpret resource files into daily notes via an agent - Implement resource file parsing with date and filename extraction logic - Add session ID computation using MD5 hash of filename - Create delete and upsert handlers for resource file operations - Add truncation and sanitization functions for tool output in auto_memory - Register auto_resource step with proper parameter validation - Add configuration for resource watch loop with file extension filters - Update default YAML config to include resource watch and digest watch loops - Add shared watch-rule logic for scan_changes and watch_changes steps - Implement foreach_dispatch and log_changes steps for change processing - Rename update_store_index_loop to index_update_loop in configuration - Refactor file chunking interface from parse to chunk method - Remove unused imports and dependencies in auto_dream step - Fix path iteration formatting in daily_index utility function - Add comprehensive integration tests for auto_resource functionality * refactor(auto_resource): format function call with multi-line parameters - Reformatted await _handle_upsert call to use multiple lines for better readability - Removed unused imports from scan_changes.py including BaseFileCatalog and ComponentEnum - Added date parameter to RuntimeContext initialization in test cases - Updated expected file paths in test assertions to include session_agent prefix - Formatted long assertion statements across multiple lines to maintain character limit - Corrected wikilink references from generic names to session_agent prefixed names	2026-06-08 16:11:32 +08:00
jinliyl	8eaa96390a	refactor(file_chunker): replace file parser with file chunker component (#276) ... * refactor(file_chunker): replace file parser with file chunker component - Rename file_parser module to file_chunker across codebase - Update BaseFileParser to BaseFileChunker with corresponding component type - Rename LinkedFileParser to MarkdownFileChunker for markdown-specific chunking - Rename ChunkedFileParser to DefaultFileChunker for default byte-based chunking - Update documentation references from file_parser to file_chunker - Modify dependency injection in BaseStep to use file_chunker instead of file_parser - Update configuration and component registration to use new chunker naming - Rename all related test files and update test assertions accordingly - Add recursive option to scan_store_changes_step in default configuration * feat(database): enhance Neo4j connection with environment variable support - Add support for NEO4J_PASSWORD environment variable as fallback - Make password parameter optional in constructor with validation - Update chromadb dependency from 1.3.5 to 1.5.7 - Configure CORS credentials based on origin settings - Import os module for environment variable access * feat(config): add timezone support and remove unused dialog directory - Added timezone field to application config with IANA timezone support - Removed unused dialog_dir configuration and related directory creation - Replaced date.today() with timezone-aware now() function across daily operations - Created evolve module with timezone-aware datetime functionality - Updated daily_create, daily_list, and daily_reindex steps to use timezone-aware dates * refactor(steps): update file chunker implementation - Replace ChunkedFileParser with DefaultFileChunker in background steps - Add module docstring to evolve steps package - Update return type annotation to reflect new chunker class usage * refactor(components): rename embedding and llm components to as_embedding and as_llm - Rename reme4/components/embedding to reme4/components/as_embedding - Rename reme4/components/llm to reme4/components/as_llm - Update all imports and references from embedding to as_embedding - Update all imports and references from llm to as_llm - Change BaseEmbedding to BaseAsEmbedding and update inheritance - Change BaseLLM to BaseAsLLM and update inheritance - Update component types from LLM/EMBEDDING to AS_LLM/AS_EMBEDDING - Update configuration keys from embedding/llm to as_embedding/as_llm - Update all property references from llm to as_llm in step classes - Update test assertions to use new component enum values * refactor(embedding_store): rename embedding parameter to as_embedding - Updated configuration key from 'embedding' to 'as_embedding' - Renamed class attribute from 'embedding' to 'as_embedding' - Updated method calls to use 'as_embedding' instead of 'embedding' - Changed parameter name in constructor from 'embedding' to 'as_embedding' - Updated documentation to reflect new parameter name - Modified health check to use 'as_embedding' property * feat(agent_wrapper): add unified agent wrapper component with multiple backends - Introduce BaseAgentWrapper abstract base class for agent implementations - Add AsAgentWrapper implementation using AgentScope framework - Add CcAgentWrapper implementation using Claude Code SDK - Register agent_wrapper component type in ComponentEnum - Configure default agent_wrapper settings in default.yaml - Implement tool integration for both AgentScope and Claude Code backends - Support fluent configuration via set_system_prompt() and add_tools() methods * feat(agent-wrapper): add structured output support for agent wrappers - Import SystemMsg in AsAgentWrapper for structured output handling - Add output_schema parameter support in AsAgentWrapper with generate_structured_output - Implement set_output_schema method in BaseAgentWrapper for chaining configuration - Add output schema support in CcAgentWrapper with JSON schema format option - Return structured output when available in CcAgentWrapper response - Refactor kwargs handling to use default values consistently across wrapper classes	2026-06-05 17:27:54 +08:00
Sen Huang	a2d76cc034	refactor(auto_dream): improve recall workflow and documentation (#272) ... * feat(file_store): add concurrency protection to LocalFileStore.dump() * feat(file_catalog): replace file_store with file_catalog in DreamStep * feat: add ChannelSink for Claude Code channel notifications * feat(auto-memory): add transcript_path support and enhance metadata	2026-06-04 19:56:50 +08:00
jinliyl	a91b08f701	Revise index descriptions in reme_design.md ... Updated the descriptions of the inverted index and vector index to clarify their implementations.	2026-06-04 16:08:37 +08:00
jinliyl	cee2c3e338	Add JSONL support, reorganize vault structure, and update design docs (#274) ... * feat(config): add jsonl support and update LLM integration tests - Added jsonl extension to supported extensions in chunked backend - Refactored LLM integration tests to use async functions instead of nested runs - Created helper functions _run_basic_chat, _run_with_tool, _run_structured_output, _run_structured_output_enum - Implemented _run_all function to execute all test scenarios sequentially - Updated main execution block to use asyncio.run with consolidated test runner - Maintained all original test functionality while improving code structure * feat(config): add dialog directory configuration and reorganize vault structure - Add new dialog_dir field for dialog memory storage - Reorder directory initialization sequence in application setup - Simplify vault_dir description in configuration schema - Update thread_pool_max_workers description to be more concise - Move resource_dir definition earlier in the configuration schema - Remove redundant text from digest_dir description * docs(reme): update design documentation with layered memory architecture - Replace quick test section with comprehensive layered memory structure - Add detailed explanation of three-tier memory organization (resource, daily, digest) - Document Obsidian-compatible Markdown format with YAML front matter - Describe four types of wikilink syntax and semantic linking features - Explain AST-aware semantic chunking for document parsing - Detail self-evolving system with auto-resource, auto-memory, and auto-dream - Document directory structure and lifecycle characteristics - Add comprehensive table showing content nature, triggers, and examples - Include semantic link extraction and knowledge graph formation processes - Describe automated indexing and relationship building workflows * docs(reme): update design documentation with simplified structure and clearer explanations - Simplified directory structure overview with cleaner formatting - Updated memory layering explanation with more concise descriptions - Improved table layouts for better readability - Clarified auto-resource, auto-memory, and auto-dream processes - Streamlined indexing and search mechanism descriptions - Enhanced component system documentation with clearer backend options - Refined job list with more precise functional descriptions - Modernized layout diagrams and process flows - Consolidated repetitive content while maintaining comprehensive coverage * docs(reme): add application scenario documentation for financial industry use case - Document comprehensive example of ReMe usage in新能源 industry research - Detail the week-long process of automatic knowledge graph construction - Explain the auto-memory and auto-dream pipeline with concrete examples - Describe the extract and integrate phases for creating wiki nodes - Illustrate cross-file linking through relates_to and derived_from predicates - Show progressive graph growth from daily sessions to complete ecosystem - Demonstrate hybrid retrieval with vector and keyword search capabilities - Provide detailed directory structure and file organization patterns - Explain the three-phase workflow: ingestion, processing, and retrieval - Document the financial analyst persona and their information management needs * style(config): fix spacing in thread_pool_max_workers field definition - Corrected spacing around description parameter in Field definition - Simplified multi-line assertion to single line in LLM integration test	2026-06-04 16:04:02 +08:00
Sen Huang	16d2d84431	feat(dream): replace digester with abstraction-layer dreamer pipeline (#264) ... * feat(dream): replace digester with abstraction-layer dreamer pipeline Reframe digest as the abstract memory layer (details stay in the daily/ resource material; digest holds principles, patterns, precedents reachable via derived_from provenance edges). Replaces the old digester with a 2-phase ReAct workflow + a daily-tick wrapper: - Phase 1 (Dreamer extract): clusters material into orthogonal memory sub-units; each sub-unit maps 1:1 to a digest node (no inner atom enumeration). Biases toward fewer / richer sub-units. - Phase 2 (Dreamer integrate per sub-unit): cross-bucket recall + exactly one write decision (CREATE / UPDATE / SKIP); UPDATE shapes surfaced explicitly (corroborate / refine / correct). - CronDreamer: scans <daily_dir>/<today>.md + <daily_dir>/<today>/** + <resource_dir>/<today>/** and runs dream_one per file. Write tools are proper subclasses of the canonical file_io WriteStep / EditStep with only path-shape + bucket + E-1 edge-conservation rules layered on top: - DigestWriteStep(WriteStep): path = <digest_dir>/<bucket>/<slug>.md, must-not-exist, schema mirrors `write` (path / name / description / content) so frontmatter lands automatically. - DigestEditStep(EditStep): body-only find-and-replace + must-exist + E-1 conservation preflight (refuses if any outbound wikilink would be dropped). Configuration: - Bucket vocabulary structured in code (tuple[{name, description}]); prompt renders the heuristic block at runtime via {buckets}. - digest_dir / daily_dir / resource_dir come from app config (not tool params); prompts use {digest_dir} placeholder. - BaseStep walks class MRO when loading prompts, so subclasses inherit parent yaml without duplication. Tooling: agentscope register_tool_function schemas now wrap in the proper {"type":"function","function":{...}} envelope. OpenAIAsLLM routes base_url through client_kwargs so non-default endpoints work. Smoke: tests4/smoke/{_dreamer_fixture.py,test_dreamer_inproc.py, test_dreamer_cli.sh} drive the end-to-end pipeline. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(dreamer): split long description string across multiple lines * refactor(dream): remove hardcoded DEFAULT_DIGEST_DIR and use app_config * docs(auto-cognition): add comprehensive design document for auto-cognition system * refactor(steps): remove deprecated digest edit/write steps * refactor(config): remove redundant LLM formatter backend configuration * refactor(dreamer): improve code formatting and line breaks * feat(auto-dream): implement three-bucket classification system for knowledge organization * feat: rename dream_today step to auto-dream and refactor extraction logic	2026-06-01 19:09:59 +08:00
jinliyl	3cb2579ff7	refactor(steps): update auto-memory (#263) ... * refactor(steps): update naming conventions in components and configuration Updated naming conventions across multiple files, changing colon-separated names to underscore-separated format, and added new step definitions along with documentation updates. Key changes: - Replaced `Synchronizer` with `AutoMemory` as the counterpart component for cold-write operations - Updated naming conventions in all related configuration files (e.g., `frontmatter:read` → `frontmatter_read`) - Added new step definitions such as `submit_slug_updates` and `auto_memory` - Updated relevant documentation - Modified log output format for improved readability * refactor(evolve): Refactor the auto-memory module and update related configurations - Remove the old slug update commit step file - Add new auto-memory planner and writer steps - Update __init__.py to export the new step classes - Modify the auto_memory configuration structure in default.yaml - Update the slug field description for clearer explanation of its purpose * up * up * refactor(tests): Move unit test directory from `tests4/unittest` to `tests4/unit` Additionally, the assertion logic in test files has been updated: direct comparisons of `payload["notes"]` have been replaced with checks verifying the presence of paths and metadata within the response content. Furthermore, some test expectations have been simplified—for example, using `count` instead of asserting against specific note lists. Specific changes include: - Updating workflow configurations to align with the new test directory structure - Modifying assertions across multiple test methods to make them more flexible and maintainable - Cleaning up and optimizing parts of the test code structure This is a comprehensive test refactoring effort aimed at improving test readability and robustness. * Refactor(steps): Update memory writing logic and optimize JSON schema structure Improved the write strategy description in `auto_memory_writer.yaml` to emphasize using `edit` over `write`. Adjusted the `json_schema` structure in `base_step.py` to support the new function definition format. Also corrected grammatical issues in the related documentation. * Fix: Improve frontend data parsing error handling and update test files Added capture and handling logic for YAML parsing exceptions, providing more detailed error messages when frontend data format issues occur. Also corrected the description text in a test file.	2026-05-29 12:07:44 +08:00
Sen Huang	8c48798164	feat(jobs): add digester step for knowledge distillation from daily notes (#261) ... * feat(jobs): add digester step for knowledge distillation from daily notes * feat: add initial implementation * refactor(jobs): remove unnecessary blank lines in digester and synchronizer * feat: add initial implementation	2026-05-28 15:17:23 +08:00
jinliyl	a4efc0f776	refactor(reme4): restructure steps packages (#258) ... * fix(bm25_index): 修正BM25索引计算中的文档长度归一化问题 修复了在计算BM25相似度时对文档长度进行不正确归一化的bug，确保所有查询都能得到准确的相关性评分。 * up * up * up * up * up * up * up * up * up * up * up * up * up * up * up * up * up * refactor(steps): Rename and adjust indexing step logic - Rename `scan_changes.py` and `reindex.py` to `clear_and_scan.py` - Update implementation details of `ScanChangesStep` and `ClearAndScanStep` - Modify the scheduling mechanism in `WatchChangesStep` - Adjust step registration and parameter configuration in config files - Update related tests to align with the new interface changes * up * feat(daily): replace daily CRUD operations with slug provisioning approach * refactor(tests): migrate CRUD step tests from HTTP server to direct LocalFileStore * up * up * up * up --------- Co-authored-by: huangsen <huangsen.huang@alibaba-inc.com>	2026-05-28 14:30:30 +08:00
Sen Huang	bb354cc580	refactor(steps): reorganize step modules and remove demo steps (#255) ... * feat(config): add comprehensive job definitions for vault operations - Add utility jobs like version, search, traverse, list, read, stat - Include file operations like move, delete, upload, download - Add daily workspace management jobs: daily_list, daily_resolve, daily_reindex - Update descriptions to reflect vault-based operations instead of working_dir - Add proper section headers and documentation for each job category refactor(steps): reorganize step modules and remove demo steps - Move steps into categorized packages: common, crud, frontmatter, daily, jobs - Remove demo steps (DemoEchoStep1, DemoEchoStep2, StreamDemoStep1, StreamDemoStep2) - Add new steps: InitStep for vault initialization, TraverseStep for graph traversal - Update __init__.py to auto-import all step modules - Organize imports by functionality (common, CRUD operations, frontmatter, daily) feat(vault): implement vault-centric file operations and configuration - Change default config to use vault_dir instead of working_dir - Add environment variable support for embedding configuration - Implement file watcher with lite backend for daily/digest directories - Update search step to use 'name' instead of 'title' from frontmatter - Create ResourceEntry schema for tracking uploaded assets docs(steps): add comprehensive documentation for all step categories - Document file-I/O split by blast radius (crud vs frontmatter packages) - Add detailed descriptions for each step category and functionality - Explain the purpose and usage patterns for different types of file operations - Provide clear parameter documentation for all new job configurations * fix(config): correct vault directory path and remove unused job configurations - Fix vault_dir from 'vaultd' to 'vault' in default configuration - Remove deprecated traverse and list job configurations - Remove unused tag tooling configurations - Remove background watch_file job configuration refactor(steps): remove unused jobs module import - Comment out jobs module import in steps/__init__.py - This removes unused synchronizer and digester step registrations refactor(tests): update import path and add pylint directive - Update ResourceEntry import from reme4.schema to reme4.schema.resource_meta - Add pylint disable directive for unused argument in test datetime mocks * efactor(steps): remove unused modules from __all__ - Remove "background" module from __all__ list - Remove "jobs" module from __all__ list - These modules were no longer being used in the steps package * feat(config): update vault directory structure and remove file watcher - Change vault_dir reference from ./vault to ./vault in CLI example - Add daily_dir, digest_dir, and resource_dir configuration options - Remove file_watcher component configuration as it's no longer needed - Update comment to reflect correct module name (reme4vault) refactor(steps): add background step and remove deprecated init step - Import and register background step module - Remove deprecated InitStep from common steps - Update __all__ export list to include background step refactor(reindex): improve reindex step to scan vault directly - Update docstring to reflect vault scanning instead of watcher sync - Replace file watcher stop/start logic with direct vault path walking - Add support for suffix filtering during reindex operation - Use index_changes job to process found files refactor(wikilink_utils): enhance inbound source lookup with link scope - Import LinkScopeEnum for proper type handling - Update get_inlinks call to use ALL scope for virtual targets - Improve documentation for reverse-index lookup behavior test(refactor): clean up test suite removing deprecated functionality - Remove test_init_job and test_demo_job unit tests - Update help job assertion to check for literal command format - Change test directory from .reme to vault in CRUD tests - Remove init and demo job calls from integration test BREAKING CHANGE: Removes file_watcher component and init step * style(steps): fix import formatting in __init__.py Add proper spacing in the background module import statement to maintain consistent code style and readability. * refactor(config): change default vault directory from vault to .reme Default dev config now points vault_dir at ./.reme so `python -m reme4 start` can be run from the repo root and exercise the full atomic-tool surface against the seeded test data. BREAKING CHANGE: The default vault directory has been changed from 'vault' to '.reme' in the configuration. * docs(reme4_report): fix markdown formatting and remove extra content * refactor(file_parser): delegate wikilink extraction to WikilinkHandler * fix(search): handle empty query case gracefully - Replace assertion with conditional check for empty query - Set response success to false when query is empty - Return error message instead of throwing assertion error - Maintain existing validation for other parameters	2026-05-25 17:52:51 +08:00
Sen Huang	7d0bec60be	feat: rename working_dir to vault_dir and update documentation (#254) ... * feat: rename working_dir to vault_dir and update documentation - Rename working_dir to vault_dir across the application - Update documentation to reflect vault_dir instead of working_dir - Change FileFrontMatter title field to name field - Update .gitignore to include vault directory - Modify file path descriptions to reference vault instead of working_dir - Update related configuration and property names accordingly * refactor(steps): rename working_path to vault_path in CRUD operations - Rename parameter from `working_path` to `vault_path` in `resolve_path` function - Update all usages in append, edit, read, and write steps to use `self.vault_path` - Update documentation comments to reflect the new parameter name - Update docstring in read.py to mention `vault_dir` instead of `vault` test(chunked_file_parser): update frontmatter field from title to name - Change frontmatter field from `title` to `name` in test cases - Update comment in background steps test to reference `vault_path` instead of `working_path` * refactor(schema): remove unused ResourceEntry import * feat(file_graph): add link scope filtering to get_inlinks/get_outlinks * feat(file-store): add scope parameter to link methods	2026-05-22 18:10:29 +08:00
jinliyl	ee94d3ec8b	docs(reme4): update report with detailed architecture sections (#252) ... - Add comprehensive Markdown kernel section covering Obsidian compatibility - Include detailed explanation of YAML front matter and wikilink formats - Document smart slicing mechanism using Markdown AST instead of fixed tokens - Explain graph indexing with bidirectional links and multiple backends - Restructure sections with proper numbering from 4 to 7 - Move Markdown kernel section to appear before self-evolution features - Add detailed explanations of auto-memory, auto-dream, and auto-link processes - Document three-way hybrid search with RRF fusion and progressive expansion - Include engineering value explanations for keyword indexing in Chinese context	2026-05-22 14:58:37 +08:00
jinliyl	71e42dbad0	refactor(reme4): replace file_watcher component with background steps pipeline (#251) ... * up * up * up * up * up * up * up * up * up * up * up * feat(config): add daily_dir configuration and background job logging - Added daily_dir setting with default value 'memory' to config - Implemented logging for background job startup events - Enhanced component start logic to handle background backend type - Updated default YAML configuration structure * refactor(file_parser): replace _get_relative_path with to_vault_relative method - Remove redundant working_dir property from base file parser - Add to_vault_relative method to base component for path resolution - Update bare_file_parser to use new to_vault_relative method - Update default_file_parser to use new to_vault_relative method - Update linked_file_parser to use new to_vault_relative method - Make working_path absolute in base_component and steps - Simplify index_changes step by removing redundant base variable - Consolidate path relative logic in single shared method * docs(reme4): update report with detailed architecture sections - Add comprehensive Markdown kernel section covering Obsidian compatibility - Include detailed explanation of YAML front matter and wikilink formats - Document smart slicing mechanism using Markdown AST instead of fixed tokens - Explain graph indexing with bidirectional links and multiple backends - Restructure sections with proper numbering from 4 to 7 - Move Markdown kernel section to appear before self-evolution features - Add detailed explanations of auto-memory, auto-dream, and auto-link processes - Document three-way hybrid search with RRF fusion and progressive expansion - Include engineering value explanations for keyword indexing in Chinese context	2026-05-22 10:26:36 +08:00
imrewce	3285934f34	create/ append/ edit steps (#249) ... * feat(core): adding create append edit steps for md crud * fix(core): changing frontmatter args scope for create step * fix(core): chang write frontmatter schema; fix edit logic; fix passing name payload to http client * refractor(steps): changing step compatibility for non-md files	2026-05-21 15:57:37 +08:00
jinliyl	0c9b8ca852	doc[reme4] (#244) ... * up * up * up * up * up * up * up * up * up * feat: implementation of the read step for reme (markdown) (#245) * feat: implementation of the read step for reme (markdown) * fix(step): markdown read step fixing pr comments * fix(step): more fixes for pr comments * further fix for better review adaptation * accept absolute path * fixing base job exception * fix(core): correct tool results directory naming (#246) - Changed directory name from 'tool_result' to 'tool_results' in documentation - Updated path variable assignment to use correct plural form 'tool_results' - Ensured consistent directory naming throughout initialization logic * fix: recent unittest inconsistency (#248) * feat: implementation of the read step for reme (markdown) * fix(step): markdown read step fixing pr comments * fix(step): more fixes for pr comments * further fix for better review adaptation * accept absolute path * fixing base job exception * fix: fix test inconsistency * up --------- Co-authored-by: imrewce <wce@pku.edu.cn>	2026-05-20 14:08:28 +08:00
imrewce	bee2648ad1	feat: implementation of the read step for reme (markdown) (#245) ... * feat: implementation of the read step for reme (markdown) * fix(step): markdown read step fixing pr comments * fix(step): more fixes for pr comments * further fix for better review adaptation * accept absolute path * fixing base job exception	2026-05-19 16:19:13 +08:00
jinliyl	fdc36a22bc	docs(cli): add comprehensive CLI commands documentation (#237) ... * docs(cli): add comprehensive CLI commands documentation - Document CLI entry point and argument parsing mechanism - Add detailed command reference with parameters and behaviors - Include usage examples for common operations like start, search, and reindex - Describe backend options and service configuration overrides - Explain local vs server-side command execution patterns - Provide table format documentation for all available actions * docs(reme_design): update CLI command documentation with detailed action descriptions - Rename section from "CLI 指令" to "基础Job" and add author attribution - Add comprehensive table documenting all available actions with parameters and behaviors - Include detailed explanations for input/output parameters, defaults, and internal workflows - Update example usage commands with proper parameter passing syntax - Add metadata information for each action including health checks and component details - Clarify the difference between local actions and server-forwarded actions - Document the new list action that intercepts at client side without forwarding to server	2026-05-17 18:37:12 +08:00
jinliyl	e411eeb4c0	dev/reme4 init merge (#236)	2026-05-17 14:14:43 +08:00