Commit graph

9 commits

Author SHA1 Message Date
jinliyl
83831ec90c
feat(core): enhance reme4 (#281)
### 1. Agent Wrapper(统一 Agent 后端抽象)
- **`base_agent_wrapper.py`**:`reply()` 返回值从 `tuple[str, Any]` 改为 `dict`(含 `session_id` / `last_message` / `result` / 可选 `structured_output`);`reply_stream()` 改为产出统一的 `StreamChunk`。废弃 `add_tools()`,改为 `add_job_tools(names: list[str])`(按名解析 BaseJob)与 `add_skills()`;新增 `_resolve_job_tools()`、`_merged_kwargs()`、`_chunk()` 辅助方法及 `project_path` / `project_skills_root` 属性。
- **`as_agent_wrapper.py`(AgentScope 后端)**:
  - 会话持久化重写:`session_path` 落地到 `<vault>/<session_dir>/agentscope/`,`_load_state` 支持 `resume` / `session_id` / `fork_session`,并做 UUID 校验(`_validate_session_id`);`_cleanup_expired_sessions` 按天数清理过期会话。
  - 新增内置工具集(`BypassAnalysisBash` + Edit/Glob/Grep/Read/Write),`BypassAnalysisBash` 绕过 AgentScope 自带 Bash 静态分析以让 permission_mode 生效;`_resolve_skills()` 把配置的 skill 暴露给后端,`_load_tool_env()` 注入项目 `.env`。
  - `_event_to_chunk()` 把 20+ 种 AgentScope 事件(Reply/Text/Thinking/Data/ToolCall/ToolResult/ModelCall/ExceedMaxIters)归一化为 `StreamChunk`。
- **`cc_agent_wrapper.py`(Claude Code SDK 后端,+551 行)**:
  - 新增 `_CcFileSessionStore`:基于 vault 的文件型会话存储,实现 append(按 uuid 去重)/ load / list / delete / list_subkeys,并对路径做 `_safe_parts` + `resolve()` 防越界校验。
  - `_build_options()`:统一构建 `ClaudeAgentOptions`,处理 skills、disallowed_tools(默认禁 `WebSearch`)、`.env` 注入、Claude Code 的 API 凭据解析(`_claude_code_api_env`,多级 base_url/api_key 回退)、`CLAUDE_CONFIG_DIR` 设置、skill 目录软链接(`_ensure_claude_skill_dir`)。
  - `_raw_event_to_chunk()` / `_message_content_to_chunks()`:把 Anthropic 流式事件(message_start/delta/stop、content_block_*)与 SDK 消息块(AssistantMessage/UserMessage/ResultMessage/RateLimitEvent)转换为统一 `StreamChunk`;跟踪 block_id/block_type/tool_call_name 做关联;处理尾部 `"success"` 误报异常的吞掉逻辑。

### 2. 统一流式协议(StreamChunk / ChunkEnum)
- **`stream_chunk.py`**:`StreamChunk` 扩展为承载 AS + CC 双后端完整信息的统一结构,新增 `session_id` / `block_id` / `tool_call_id` / `tool_call_name` / `media_type` / `input_tokens` / `output_tokens` 等字段,纯文本流仍保持轻量。
- **`chunk_enum.py`**:补全生命周期标记 `REPLY_START` / `REPLY_END`,并文档化两套后端事件 → ChunkEnum 的映射。

### 3. Index 模块重构(变化批次化 + dispatch)
- 新增 `_change_batch.py`:`coalesce_changes()` 把同路径多次事件折叠为最终状态(结合 path 存在性判定),`bucket_changes()` 按 watchfiles.Change 分桶。
- 新增 `init_changes.py`(`InitChangesStep`):一次性扫描,对比 file_store / file_catalog 已索引节点计算 added/modified/deleted,写入 `context["changes"]` 后 dispatch。
- 新增 `update_changes.py`:抽象基类 `ChangeApplyStep` 统一 added/modified/deleted 处理与错误收集;`UpdateCatalogStep`(写 file_catalog)、`UpdateIndexStep`(写 file_store,含按后缀解析 chunker)。
- **`watch_changes.py`**:改用 `dispatch_step_specs`(基类提供的 `dispatch_steps()`),每批先 `coalesce_changes` 再 dispatch;默认参数调整(debounce 5000ms / step 1000ms / poll 5000ms)并暴露常量。
- 删除旧步骤:`clear_and_scan` / `foreach_dispatch` / `scan_changes` / `update_catalog`(旧) / `update_index`(旧);`clear_store.py` 取代 clear_and_scan。

### 4. Evolve / Dream 模块(拆分为多步 pipeline)
- 删除旧的单体 `auto_dream.py` / `dream.py` / `dream.yaml`,新增 `dream/` 子包,按 5 个步骤组织:
  - **`extract.py`**:扫描当日 day-index + daily 笔记,对比 file_catalog 找出 changed/deleted,调用 LLM 全局抽取 `units`(procedure/personal/wiki 三桶)与 `topics`,路径与桶做清洗/路由。
  - **`integrate.py`**:逐个 unit 调用 LLM 写入 digest,结构化输出 `IntegrateOutcome`(CREATE/CORROBORATE/REFINE/CORRECT),失败 unit/路径收集回写。
  - **`topics.py`**:写 `daily/<date>/interests.yaml`,结合当天已有 + 近 N 天做去重(`normalize_topic`),可走 LLM 或纯规则去重两条路径。
  - **`proactive.py`**:读取当日 `interests.yaml`,作为主动推荐话题的入口。
  - **`finish.py`**:把变更路径落盘到 dream file_catalog(checkpoint),渲染最终汇总摘要。
- 新增 `schema.py`(`DreamState` 等跨步骤共享状态与结构化输出模型)与 `utils.py`(状态存取、扫描打包、YAML 读写、结构化回复解析等公共函数)。
- `evolve/__init__.py` 导出全部新 step。

### 5. auto_memory / auto_resource(适配新 Agent API)
- **`auto_memory.py`**:会话路径迁移到 `<session_dir>/dialog/<session_id>.jsonl`;改用 `job_tools`;新增 `source_conversation` frontmatter 反向链接(`_session_link`);执行后刷新 day 索引(`refresh_day_index`),并对 session_id 做合法性校验。
- **`auto_resource.py`**:资源改用「同名 daily note」方案(`_compute_note_stem` 取文件 stem);批量处理 `changes: list[dict]`(`_handle_change` 逐项处理,返回逐项结果摘要);agent 会话 id 用稳定的 `uuid5`;同样刷新 day 索引。

### 6. BaseStep 基类增强
- 新增 `dispatch_steps` / `dispatch_step_specs` 机制:`_resolve_dispatch_step()` 支持字符串或 dict 形式的 step spec,`dispatch_steps()` 复用当前 context 调用下游 step。
- 新增 `config_value()`:按 key 取 app config,缺失时回退 `ApplicationConfig` 默认值。
- 小幅清理:`language` 初始化、`copy()`、`Ref.__init__` 签名精简。

### 7. Components 改动
- **`file_store/local_file_store.py`**:持久化改用 zstd 压缩(`.jsonl.zst`,通过新 `utils/jsonl_zst.py`);upsert 时先删除旧 chunk 的 keyword 文档;embedding 复用改为 `(text, embedding)` 键控,要求文本一致才复用;新增 `_matches_search_filter()` 对 vector/keyword 搜索做 path/path_prefix/metadata 的统一后过滤。
- **`keyword_index/bm25_index.py`**:索引文件名加入组件名 + tokenizer 指纹(sha256 前 12 位),快照/恢复时校验指纹防配置漂移;空索引 dump 时删除文件,加载失败抛错而非静默。
- **`file_chunker/markdown_file_chunker.py`**:弃用 `python-frontmatter`,改用内置 YAML 解析(非法 YAML 不阻断正文索引),并修正因 frontmatter 占用行号导致的 AST 行号偏移(`line_offset`)。
- **`cron_job.py`**:大幅简化(-187 行),由原来「dispatch 外部 job/step + 多种调度模式」改为「在自身 steps 上跑 cron 表达式」;`Application` 启动顺序随之调整为 base > stream > background > cron。
- 其余小调整:service(base/http/mcp)、file_graph、file_catalog、as_llm、as_embedding、tokenizer、prompt_handler、base_component 的签名/接口微调。

### 8. Application 生命周期
- `_start()` 启动顺序明确为 components → base → stream → background → cron,启动失败会触发 `_close()` 回滚并 re-raise(不再吞异常)。
- 启动时创建 `session_dir` 目录;新增 `update_component()`(按类型/名就地更新已存在组件,不存在则报错)。

### 9. File IO / 路径安全
- **`_path.py`**:`resolve_path` 增加 vault 越界防护(`is_relative_to` 校验),禁止 `.` / `..` 路径分量,支持 `allow_empty`。
- **`read.py`**:大文件(超过 `MAX_FILE_READ_BYTES`)走按行读取 `read_file_lines_safe`,避免一次性载入内存。
- **`_file_io.py` / `_daily_index.py` / `_path.py`** 等支持函数补齐(如 `refresh_day_index`、`read_file_lines_safe`)。
- **`env_utils.py`**:新增 `parse_env_file()`,`load_env()` 返回加载到的键值、支持 `override`、对无路径调用做幂等缓存。

### 10. Config
- `ApplicationConfig` 新增 `session_dir`(默认 `reme_session`)。
- `config_parser.py`:环境变量展开后做类型转换(`_convert_value`)、dot-notation 与 key=value 参数校验更严格、配置文件路径支持相对 `_CONFIG_DIR` 查找、根非 dict 报错。
- `default.yaml`:作业编排改用 `init_changes_step` + `dispatch_steps`(index/resource/digest 三个 watch loop 与 reindex);新增 `auto_dream`(4 步)、`proactive` 作业,移除旧 `dream`;file_catalog 增配 `resource` / `digest` / `dream` 实例;LLM 默认值与 Claude Code 凭据配置调整(tool_result_limit 50000、thinking_enable=false 等)。

### 11. 其它
- 新增 `steps/common/add.py`(`AddStep` 算术 demo)、`channel/__init__.py` 与 common `__init__` 导出整理。
- 新增 4 篇文档:`docs4/auto_dream_logic_and_step_refactor.md`、`docs4/watch_loop_step_refactor_plan.md`、`docs4/todo.md`,以及 `reme_design.md` 更新。
**
2026-06-19 01:35:31 +08:00
jinliyl
8eaa96390a
refactor(file_chunker): replace file parser with file chunker component (#276)
Some checks failed
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
Tests ReMe / Unit Tests - py3.11 (push) Has been cancelled
Tests ReMe / Unit Tests - py3.12 (push) Has been cancelled
Tests ReMe / Unit Tests - py3.13 (push) Has been cancelled
* refactor(file_chunker): replace file parser with file chunker component

- Rename file_parser module to file_chunker across codebase
- Update BaseFileParser to BaseFileChunker with corresponding component type
- Rename LinkedFileParser to MarkdownFileChunker for markdown-specific chunking
- Rename ChunkedFileParser to DefaultFileChunker for default byte-based chunking
- Update documentation references from file_parser to file_chunker
- Modify dependency injection in BaseStep to use file_chunker instead of file_parser
- Update configuration and component registration to use new chunker naming
- Rename all related test files and update test assertions accordingly
- Add recursive option to scan_store_changes_step in default configuration

* feat(database): enhance Neo4j connection with environment variable support

- Add support for NEO4J_PASSWORD environment variable as fallback
- Make password parameter optional in constructor with validation
- Update chromadb dependency from 1.3.5 to 1.5.7
- Configure CORS credentials based on origin settings
- Import os module for environment variable access

* feat(config): add timezone support and remove unused dialog directory

- Added timezone field to application config with IANA timezone support
- Removed unused dialog_dir configuration and related directory creation
- Replaced date.today() with timezone-aware now() function across daily operations
- Created evolve module with timezone-aware datetime functionality
- Updated daily_create, daily_list, and daily_reindex steps to use timezone-aware dates

* refactor(steps): update file chunker implementation

- Replace ChunkedFileParser with DefaultFileChunker in background steps
- Add module docstring to evolve steps package
- Update return type annotation to reflect new chunker class usage

* refactor(components): rename embedding and llm components to as_embedding and as_llm

- Rename reme4/components/embedding to reme4/components/as_embedding
- Rename reme4/components/llm to reme4/components/as_llm
- Update all imports and references from embedding to as_embedding
- Update all imports and references from llm to as_llm
- Change BaseEmbedding to BaseAsEmbedding and update inheritance
- Change BaseLLM to BaseAsLLM and update inheritance
- Update component types from LLM/EMBEDDING to AS_LLM/AS_EMBEDDING
- Update configuration keys from embedding/llm to as_embedding/as_llm
- Update all property references from llm to as_llm in step classes
- Update test assertions to use new component enum values

* refactor(embedding_store): rename embedding parameter to as_embedding

- Updated configuration key from 'embedding' to 'as_embedding'
- Renamed class attribute from 'embedding' to 'as_embedding'
- Updated method calls to use 'as_embedding' instead of 'embedding'
- Changed parameter name in constructor from 'embedding' to 'as_embedding'
- Updated documentation to reflect new parameter name
- Modified health check to use 'as_embedding' property

* feat(agent_wrapper): add unified agent wrapper component with multiple backends

- Introduce BaseAgentWrapper abstract base class for agent implementations
- Add AsAgentWrapper implementation using AgentScope framework
- Add CcAgentWrapper implementation using Claude Code SDK
- Register agent_wrapper component type in ComponentEnum
- Configure default agent_wrapper settings in default.yaml
- Implement tool integration for both AgentScope and Claude Code backends
- Support fluent configuration via set_system_prompt() and add_tools() methods

* feat(agent-wrapper): add structured output support for agent wrappers

- Import SystemMsg in AsAgentWrapper for structured output handling
- Add output_schema parameter support in AsAgentWrapper with generate_structured_output
- Implement set_output_schema method in BaseAgentWrapper for chaining configuration
- Add output schema support in CcAgentWrapper with JSON schema format option
- Return structured output when available in CcAgentWrapper response
- Refactor kwargs handling to use default values consistently across wrapper classes
2026-06-05 17:27:54 +08:00
jinliyl
041f957a7f
refactor(components) components and file I/O, fix method calls and validation (#268)
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.10 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
* refactor(components): extract shared component state into mixin

- Introduce ComponentMixin class with shared state for components and steps
- Move identity, config, and vault path functionality to ComponentMixin
- Update BaseComponent to inherit from ComponentMixin
- Update BaseStep to inherit from ComponentMixin
- Consolidate vault path helper methods in ComponentMixin
- Remove duplicate vault path implementations from BaseComponent and BaseStep
- Add ComponentMixin to components module exports

* refactor(file_io): implement path locks cache eviction mechanism

- Add _PATH_LOCKS_MAX constant set to 1024 for cache size limit
- Implement cache eviction logic when locks exceed maximum capacity
- Remove half of unlocked entries when cache limit is reached
- Use list comprehension to identify unlocked locks for removal
- Maintain existing path normalization and locking behavior

fix(edit): correct method call from public to private fail method

- Change self.fail to self._fail for internal error handling
- Maintain consistent private method usage within class

fix(mcp_client): change pop to get for optional command and args

- Replace kwargs.pop with kwargs.get to avoid removing keys
- Preserve original kwargs dictionary contents
- Maintain default empty string and list values

feat(reme): add client backend validation with error raising

- Check if client_cls is None before instantiation
- Raise ValueError with descriptive message for unknown backends
- Provide clear error feedback for invalid backend configurations

* fix(components): move directory creation to start method

- Moved component_metadata_path.mkdir call from __init__ to _start in base_keyword_index
- Moved component_metadata_path.mkdir call from __init__ to _start in local_file_graph
- Moved component_metadata_path.mkdir call from __init__ to _start in local_file_store
- Ensures directory creation happens after component initialization
- Prevents potential issues with path creation during object construction

* fix(steps): replace assertions with runtime errors for app_context validation

- Replace assert statements with explicit RuntimeError exceptions when app_context is None
- Add descriptive error messages for better debugging when resolving components
- Replace assert in resolve_component method with proper exception handling
- Replace assert in get_file_parser method with proper exception handling
- Maintain same functionality while improving error reporting clarity

* refactor(file_io): split file IO utilities into modular components

- Move daily note helpers to separate _daily_index module
- Extract path validation and resolution to new _path module
- Remove unused code and imports from _file_io module
- Update import statements across affected modules
- Introduce WikilinkHandler utility for link parsing
- Replace regex-based link extraction with WikilinkHandler
- Add integration JSONL files to gitignore
- Consolidate file locking mechanism in _file_io module

* style(formatter): fix spacing issues in file IO and chunked file parser

- Fixed whitespace around colon in slice notation in file_io.py
- Corrected spacing around colon in slice notation in chunked_file_parser.py
- Applied consistent formatting for array slicing operations
- Improved code readability by standardizing space placement in ranges

* refactor(steps): replace property-based component resolution with Ref descriptor

- Introduce Ref descriptor class for lazy component dependency resolution
- Replace _resolve method and individual properties with Ref descriptors
- Add as_llm, as_llm_formatter, as_token_counter, file_store, and embedding Ref attributes
- Remove legacy property methods and resolve logic from BaseStep
- Add cache clearing mechanism for Ref values during step calls
- Update UpdateCatalogStep to use Ref instead of property-based resolution
2026-06-01 11:35:19 +08:00
jinliyl
a4efc0f776
refactor(reme4): restructure steps packages (#258)
* fix(bm25_index): 修正BM25索引计算中的文档长度归一化问题

修复了在计算BM25相似度时对文档长度进行不正确归一化的bug,确保所有查询都能得到准确的相关性评分。

* up

* up

* up

* up

* up

* up

* up

* up

* up

* up

* up

* up

* up

* up

* up

* up

* up

* refactor(steps): Rename and adjust indexing step logic

- Rename `scan_changes.py` and `reindex.py` to `clear_and_scan.py`
- Update implementation details of `ScanChangesStep` and `ClearAndScanStep`
- Modify the scheduling mechanism in `WatchChangesStep`
- Adjust step registration and parameter configuration in config files
- Update related tests to align with the new interface changes

* up

* feat(daily): replace daily CRUD operations with slug provisioning approach

* refactor(tests): migrate CRUD step tests from HTTP server to direct LocalFileStore

* up

* up

* up

* up

---------

Co-authored-by: huangsen <huangsen.huang@alibaba-inc.com>
2026-05-28 14:30:30 +08:00
Sen Huang
bb354cc580
refactor(steps): reorganize step modules and remove demo steps (#255)
* feat(config): add comprehensive job definitions for vault operations

- Add utility jobs like version, search, traverse, list, read, stat
- Include file operations like move, delete, upload, download
- Add daily workspace management jobs: daily_list, daily_resolve, daily_reindex
- Update descriptions to reflect vault-based operations instead of working_dir
- Add proper section headers and documentation for each job category

refactor(steps): reorganize step modules and remove demo steps

- Move steps into categorized packages: common, crud, frontmatter, daily, jobs
- Remove demo steps (DemoEchoStep1, DemoEchoStep2, StreamDemoStep1, StreamDemoStep2)
- Add new steps: InitStep for vault initialization, TraverseStep for graph traversal
- Update __init__.py to auto-import all step modules
- Organize imports by functionality (common, CRUD operations, frontmatter, daily)

feat(vault): implement vault-centric file operations and configuration

- Change default config to use vault_dir instead of working_dir
- Add environment variable support for embedding configuration
- Implement file watcher with lite backend for daily/digest directories
- Update search step to use 'name' instead of 'title' from frontmatter
- Create ResourceEntry schema for tracking uploaded assets

docs(steps): add comprehensive documentation for all step categories

- Document file-I/O split by blast radius (crud vs frontmatter packages)
- Add detailed descriptions for each step category and functionality
- Explain the purpose and usage patterns for different types of file operations
- Provide clear parameter documentation for all new job configurations

* fix(config): correct vault directory path and remove unused job configurations

- Fix vault_dir from 'vaultd' to 'vault' in default configuration
- Remove deprecated traverse and list job configurations
- Remove unused tag tooling configurations
- Remove background watch_file job configuration

refactor(steps): remove unused jobs module import

- Comment out jobs module import in steps/__init__.py
- This removes unused synchronizer and digester step registrations

refactor(tests): update import path and add pylint directive

- Update ResourceEntry import from reme4.schema to reme4.schema.resource_meta
- Add pylint disable directive for unused argument in test datetime mocks

* efactor(steps): remove unused modules from __all__

- Remove "background" module from __all__ list
- Remove "jobs" module from __all__ list
- These modules were no longer being used in the steps package

* feat(config): update vault directory structure and remove file watcher

- Change vault_dir reference from ./vault to ./vault in CLI example
- Add daily_dir, digest_dir, and resource_dir configuration options
- Remove file_watcher component configuration as it's no longer needed
- Update comment to reflect correct module name (reme4vault)

refactor(steps): add background step and remove deprecated init step

- Import and register background step module
- Remove deprecated InitStep from common steps
- Update __all__ export list to include background step

refactor(reindex): improve reindex step to scan vault directly

- Update docstring to reflect vault scanning instead of watcher sync
- Replace file watcher stop/start logic with direct vault path walking
- Add support for suffix filtering during reindex operation
- Use index_changes job to process found files

refactor(wikilink_utils): enhance inbound source lookup with link scope

- Import LinkScopeEnum for proper type handling
- Update get_inlinks call to use ALL scope for virtual targets
- Improve documentation for reverse-index lookup behavior

test(refactor): clean up test suite removing deprecated functionality

- Remove test_init_job and test_demo_job unit tests
- Update help job assertion to check for literal command format
- Change test directory from .reme to vault in CRUD tests
- Remove init and demo job calls from integration test

BREAKING CHANGE: Removes file_watcher component and init step

* style(steps): fix import formatting in __init__.py

Add proper spacing in the background module import statement
to maintain consistent code style and readability.

* refactor(config): change default vault directory from vault to .reme

Default dev config now points vault_dir at ./.reme so `python -m
reme4 start` can be run from the repo root and exercise the full
atomic-tool surface against the seeded test data.

BREAKING CHANGE: The default vault directory has been changed from
'vault' to '.reme' in the configuration.

* docs(reme4_report): fix markdown formatting and remove extra content

* refactor(file_parser): delegate wikilink extraction to WikilinkHandler

* fix(search): handle empty query case gracefully

- Replace assertion with conditional check for empty query
- Set response success to false when query is empty
- Return error message instead of throwing assertion error
- Maintain existing validation for other parameters
2026-05-25 17:52:51 +08:00
Sen Huang
7d0bec60be
feat: rename working_dir to vault_dir and update documentation (#254)
Some checks failed
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
Tests ReMe / Unit Tests - py3.10 (push) Has been cancelled
Tests ReMe / Unit Tests - py3.13 (push) Has been cancelled
* feat: rename working_dir to vault_dir and update documentation

- Rename working_dir to vault_dir across the application
- Update documentation to reflect vault_dir instead of working_dir
- Change FileFrontMatter title field to name field
- Update .gitignore to include vault directory
- Modify file path descriptions to reference vault instead of working_dir
- Update related configuration and property names accordingly

* refactor(steps): rename working_path to vault_path in CRUD operations

- Rename parameter from `working_path` to `vault_path` in `resolve_path` function
- Update all usages in append, edit, read, and write steps to use `self.vault_path`
- Update documentation comments to reflect the new parameter name
- Update docstring in read.py to mention `vault_dir` instead of `vault`

test(chunked_file_parser): update frontmatter field from title to name

- Change frontmatter field from `title` to `name` in test cases
- Update comment in background steps test to reference `vault_path` instead of `working_path`

* refactor(schema): remove unused ResourceEntry import

* feat(file_graph): add link scope filtering to get_inlinks/get_outlinks

* feat(file-store): add scope parameter to link methods
2026-05-22 18:10:29 +08:00
jinliyl
71e42dbad0
refactor(reme4): replace file_watcher component with background steps pipeline (#251)
* up

* up

* up

* up

* up

* up

* up

* up

* up

* up

* up

* feat(config): add daily_dir configuration and background job logging

- Added daily_dir setting with default value 'memory' to config
- Implemented logging for background job startup events
- Enhanced component start logic to handle background backend type
- Updated default YAML configuration structure

* refactor(file_parser): replace _get_relative_path with to_vault_relative method

- Remove redundant working_dir property from base file parser
- Add to_vault_relative method to base component for path resolution
- Update bare_file_parser to use new to_vault_relative method
- Update default_file_parser to use new to_vault_relative method
- Update linked_file_parser to use new to_vault_relative method
- Make working_path absolute in base_component and steps
- Simplify index_changes step by removing redundant base variable
- Consolidate path relative logic in single shared method

* docs(reme4): update report with detailed architecture sections

- Add comprehensive Markdown kernel section covering Obsidian compatibility
- Include detailed explanation of YAML front matter and wikilink formats
- Document smart slicing mechanism using Markdown AST instead of fixed tokens
- Explain graph indexing with bidirectional links and multiple backends
- Restructure sections with proper numbering from 4 to 7
- Move Markdown kernel section to appear before self-evolution features
- Add detailed explanations of auto-memory, auto-dream, and auto-link processes
- Document three-way hybrid search with RRF fusion and progressive expansion
- Include engineering value explanations for keyword indexing in Chinese context
2026-05-22 10:26:36 +08:00
Sen Huang
357415dd49
feat: add Neo4j file graph support and markdown parser with wikilink extraction (#240)
* feat: add Neo4j file graph support and markdown parser with wikilink extraction

- Add Neo4jFileGraph implementation for property-graph storage with
  virtual/real node handling and link management
- Introduce LinkedFileParser for markdown files with frontmatter,
  wikilink graph extraction, and full-skeleton chunking
- Update pyproject.toml to include pyyaml, mistletoe, and neo4j
  dependencies
- Modify .gitignore to exclude /vault and structure.md
- Change reme CLI entry point from reme_ai.main to remecli.reme
- Register new neo4j and md components in respective registries

* refactor(file-graph): add chunk_ids support to Neo4jFileGraph

Add chunk_ids field to File node properties in Neo4jFileGraph to
enable better content chunk tracking and management.

BREAKING CHANGE: File node schema now includes chunk_ids property
which may affect existing integrations.

feat(parser): implement wikilink resolution logic

Move path resolution logic from utils/path_resolver to
linked_file_parser module and enhance wikilink resolution with
folder-note rule support and improved error handling.

fix(tests): update test assertions and variable names

Update test cases to reflect changes in data structures and
variable naming conventions across various components.

chore(config): update package entry point reference

Change reme CLI entry point from remecli.reme:main to
reme_ai.reme:main in pyproject.toml.

refactor(utils): remove deprecated path_resolver module

Remove the old path_resolver utility module as its functionality
has been moved to linked_file_parser.

docs(file-graph): update Neo4jFileGraph documentation

Update class docstrings and comments to reflect new chunk_ids
property and other structural changes.

style(formatting): adjust code formatting and line breaks

Minor formatting improvements including line length optimization
and consistent spacing adjustments throughout the codebase.

* fix(pyproject.toml): correct entry point for reme command

Change the entry point from "reme_ai.reme:main" to "reme_ai.main:main"
to fix the module reference for the reme command in project scripts.
2026-05-18 14:25:14 +08:00
jinliyl
e411eeb4c0
dev/reme4 init merge (#236) 2026-05-17 14:14:43 +08:00