Commit graph

4 commits

Author SHA1 Message Date
Sen Huang
bb354cc580
refactor(steps): reorganize step modules and remove demo steps (#255)
* feat(config): add comprehensive job definitions for vault operations

- Add utility jobs like version, search, traverse, list, read, stat
- Include file operations like move, delete, upload, download
- Add daily workspace management jobs: daily_list, daily_resolve, daily_reindex
- Update descriptions to reflect vault-based operations instead of working_dir
- Add proper section headers and documentation for each job category

refactor(steps): reorganize step modules and remove demo steps

- Move steps into categorized packages: common, crud, frontmatter, daily, jobs
- Remove demo steps (DemoEchoStep1, DemoEchoStep2, StreamDemoStep1, StreamDemoStep2)
- Add new steps: InitStep for vault initialization, TraverseStep for graph traversal
- Update __init__.py to auto-import all step modules
- Organize imports by functionality (common, CRUD operations, frontmatter, daily)

feat(vault): implement vault-centric file operations and configuration

- Change default config to use vault_dir instead of working_dir
- Add environment variable support for embedding configuration
- Implement file watcher with lite backend for daily/digest directories
- Update search step to use 'name' instead of 'title' from frontmatter
- Create ResourceEntry schema for tracking uploaded assets

docs(steps): add comprehensive documentation for all step categories

- Document file-I/O split by blast radius (crud vs frontmatter packages)
- Add detailed descriptions for each step category and functionality
- Explain the purpose and usage patterns for different types of file operations
- Provide clear parameter documentation for all new job configurations

* fix(config): correct vault directory path and remove unused job configurations

- Fix vault_dir from 'vaultd' to 'vault' in default configuration
- Remove deprecated traverse and list job configurations
- Remove unused tag tooling configurations
- Remove background watch_file job configuration

refactor(steps): remove unused jobs module import

- Comment out jobs module import in steps/__init__.py
- This removes unused synchronizer and digester step registrations

refactor(tests): update import path and add pylint directive

- Update ResourceEntry import from reme4.schema to reme4.schema.resource_meta
- Add pylint disable directive for unused argument in test datetime mocks

* efactor(steps): remove unused modules from __all__

- Remove "background" module from __all__ list
- Remove "jobs" module from __all__ list
- These modules were no longer being used in the steps package

* feat(config): update vault directory structure and remove file watcher

- Change vault_dir reference from ./vault to ./vault in CLI example
- Add daily_dir, digest_dir, and resource_dir configuration options
- Remove file_watcher component configuration as it's no longer needed
- Update comment to reflect correct module name (reme4vault)

refactor(steps): add background step and remove deprecated init step

- Import and register background step module
- Remove deprecated InitStep from common steps
- Update __all__ export list to include background step

refactor(reindex): improve reindex step to scan vault directly

- Update docstring to reflect vault scanning instead of watcher sync
- Replace file watcher stop/start logic with direct vault path walking
- Add support for suffix filtering during reindex operation
- Use index_changes job to process found files

refactor(wikilink_utils): enhance inbound source lookup with link scope

- Import LinkScopeEnum for proper type handling
- Update get_inlinks call to use ALL scope for virtual targets
- Improve documentation for reverse-index lookup behavior

test(refactor): clean up test suite removing deprecated functionality

- Remove test_init_job and test_demo_job unit tests
- Update help job assertion to check for literal command format
- Change test directory from .reme to vault in CRUD tests
- Remove init and demo job calls from integration test

BREAKING CHANGE: Removes file_watcher component and init step

* style(steps): fix import formatting in __init__.py

Add proper spacing in the background module import statement
to maintain consistent code style and readability.

* refactor(config): change default vault directory from vault to .reme

Default dev config now points vault_dir at ./.reme so `python -m
reme4 start` can be run from the repo root and exercise the full
atomic-tool surface against the seeded test data.

BREAKING CHANGE: The default vault directory has been changed from
'vault' to '.reme' in the configuration.

* docs(reme4_report): fix markdown formatting and remove extra content

* refactor(file_parser): delegate wikilink extraction to WikilinkHandler

* fix(search): handle empty query case gracefully

- Replace assertion with conditional check for empty query
- Set response success to false when query is empty
- Return error message instead of throwing assertion error
- Maintain existing validation for other parameters
2026-05-25 17:52:51 +08:00
jinliyl
24cff10d46
feat(file_store): add FAISS-backed local file store implementation (#253)
* feat(file_store): add FAISS-backed local file store implementation

- Introduce FaissLocalFileStore class with vector search capabilities using FAISS IndexFlatIP
- Implement FAISS index persistence with binary format and JSON id-map sidecar
- Add automatic index rebuilding when sidecar files are missing or corrupted
- Support tombstone mechanism for efficient deletion and compaction
- Register 'faiss' component type in the registry system
- Add faiss-cpu dependency requirement to pyproject.toml
- Update configuration schema to use simplified parameter structure
- Enhance search step to support parameter override from runtime context
- Add comprehensive unit tests for FAISS store functionality
- Implement fallback to parent methods for basic CRUD operations

* refactor(search): simplify parameter retrieval logic

- Removed _param method that checked context and kwargs
- Directly use self.kwargs.get for all parameter retrievals
- Maintained same default values for vector_weight, candidate_multiplier, expand_links, and max_links_per_direction
- Reduced code complexity by eliminating redundant context checking logic
2026-05-22 16:14:38 +08:00
jinliyl
71e42dbad0
refactor(reme4): replace file_watcher component with background steps pipeline (#251)
* up

* up

* up

* up

* up

* up

* up

* up

* up

* up

* up

* feat(config): add daily_dir configuration and background job logging

- Added daily_dir setting with default value 'memory' to config
- Implemented logging for background job startup events
- Enhanced component start logic to handle background backend type
- Updated default YAML configuration structure

* refactor(file_parser): replace _get_relative_path with to_vault_relative method

- Remove redundant working_dir property from base file parser
- Add to_vault_relative method to base component for path resolution
- Update bare_file_parser to use new to_vault_relative method
- Update default_file_parser to use new to_vault_relative method
- Update linked_file_parser to use new to_vault_relative method
- Make working_path absolute in base_component and steps
- Simplify index_changes step by removing redundant base variable
- Consolidate path relative logic in single shared method

* docs(reme4): update report with detailed architecture sections

- Add comprehensive Markdown kernel section covering Obsidian compatibility
- Include detailed explanation of YAML front matter and wikilink formats
- Document smart slicing mechanism using Markdown AST instead of fixed tokens
- Explain graph indexing with bidirectional links and multiple backends
- Restructure sections with proper numbering from 4 to 7
- Move Markdown kernel section to appear before self-evolution features
- Add detailed explanations of auto-memory, auto-dream, and auto-link processes
- Document three-way hybrid search with RRF fusion and progressive expansion
- Include engineering value explanations for keyword indexing in Chinese context
2026-05-22 10:26:36 +08:00
jinliyl
e411eeb4c0
dev/reme4 init merge (#236) 2026-05-17 14:14:43 +08:00