Reframe digest as the abstract memory layer (details stay in the daily/
resource material; digest holds principles, patterns, precedents reachable
via derived_from provenance edges). Replaces the old digester with a
2-phase ReAct workflow + a daily-tick wrapper:
- Phase 1 (Dreamer extract): clusters material into orthogonal memory
sub-units; each sub-unit maps 1:1 to a digest node (no inner atom
enumeration). Biases toward fewer / richer sub-units.
- Phase 2 (Dreamer integrate per sub-unit): cross-bucket recall +
exactly one write decision (CREATE / UPDATE / SKIP); UPDATE shapes
surfaced explicitly (corroborate / refine / correct).
- CronDreamer: scans <daily_dir>/<today>.md + <daily_dir>/<today>/**
+ <resource_dir>/<today>/** and runs dream_one per file.
Write tools are proper subclasses of the canonical file_io WriteStep /
EditStep with only path-shape + bucket + E-1 edge-conservation rules
layered on top:
- DigestWriteStep(WriteStep): path = <digest_dir>/<bucket>/<slug>.md,
must-not-exist, schema mirrors `write` (path / name / description /
content) so frontmatter lands automatically.
- DigestEditStep(EditStep): body-only find-and-replace + must-exist +
E-1 conservation preflight (refuses if any outbound wikilink would
be dropped).
Configuration:
- Bucket vocabulary structured in code (tuple[{name, description}]);
prompt renders the heuristic block at runtime via {buckets}.
- digest_dir / daily_dir / resource_dir come from app config (not tool
params); prompts use {digest_dir} placeholder.
- BaseStep walks class MRO when loading prompts, so subclasses inherit
parent yaml without duplication.
Tooling: agentscope register_tool_function schemas now wrap in the
proper {"type":"function","function":{...}} envelope. OpenAIAsLLM
routes base_url through client_kwargs so non-default endpoints work.
Smoke: tests4/smoke/{_dreamer_fixture.py,test_dreamer_inproc.py,
test_dreamer_cli.sh} drive the end-to-end pipeline.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(components): extract shared component state into mixin
- Introduce ComponentMixin class with shared state for components and steps
- Move identity, config, and vault path functionality to ComponentMixin
- Update BaseComponent to inherit from ComponentMixin
- Update BaseStep to inherit from ComponentMixin
- Consolidate vault path helper methods in ComponentMixin
- Remove duplicate vault path implementations from BaseComponent and BaseStep
- Add ComponentMixin to components module exports
* refactor(file_io): implement path locks cache eviction mechanism
- Add _PATH_LOCKS_MAX constant set to 1024 for cache size limit
- Implement cache eviction logic when locks exceed maximum capacity
- Remove half of unlocked entries when cache limit is reached
- Use list comprehension to identify unlocked locks for removal
- Maintain existing path normalization and locking behavior
fix(edit): correct method call from public to private fail method
- Change self.fail to self._fail for internal error handling
- Maintain consistent private method usage within class
fix(mcp_client): change pop to get for optional command and args
- Replace kwargs.pop with kwargs.get to avoid removing keys
- Preserve original kwargs dictionary contents
- Maintain default empty string and list values
feat(reme): add client backend validation with error raising
- Check if client_cls is None before instantiation
- Raise ValueError with descriptive message for unknown backends
- Provide clear error feedback for invalid backend configurations
* fix(components): move directory creation to start method
- Moved component_metadata_path.mkdir call from __init__ to _start in base_keyword_index
- Moved component_metadata_path.mkdir call from __init__ to _start in local_file_graph
- Moved component_metadata_path.mkdir call from __init__ to _start in local_file_store
- Ensures directory creation happens after component initialization
- Prevents potential issues with path creation during object construction
* fix(steps): replace assertions with runtime errors for app_context validation
- Replace assert statements with explicit RuntimeError exceptions when app_context is None
- Add descriptive error messages for better debugging when resolving components
- Replace assert in resolve_component method with proper exception handling
- Replace assert in get_file_parser method with proper exception handling
- Maintain same functionality while improving error reporting clarity
* refactor(file_io): split file IO utilities into modular components
- Move daily note helpers to separate _daily_index module
- Extract path validation and resolution to new _path module
- Remove unused code and imports from _file_io module
- Update import statements across affected modules
- Introduce WikilinkHandler utility for link parsing
- Replace regex-based link extraction with WikilinkHandler
- Add integration JSONL files to gitignore
- Consolidate file locking mechanism in _file_io module
* style(formatter): fix spacing issues in file IO and chunked file parser
- Fixed whitespace around colon in slice notation in file_io.py
- Corrected spacing around colon in slice notation in chunked_file_parser.py
- Applied consistent formatting for array slicing operations
- Improved code readability by standardizing space placement in ranges
* refactor(steps): replace property-based component resolution with Ref descriptor
- Introduce Ref descriptor class for lazy component dependency resolution
- Replace _resolve method and individual properties with Ref descriptors
- Add as_llm, as_llm_formatter, as_token_counter, file_store, and embedding Ref attributes
- Remove legacy property methods and resolve logic from BaseStep
- Add cache clearing mechanism for Ref values during step calls
- Update UpdateCatalogStep to use Ref instead of property-based resolution
* refactor(steps): update naming conventions in components and configuration
Updated naming conventions across multiple files, changing colon-separated names to underscore-separated format, and added new step definitions along with documentation updates.
Key changes:
- Replaced `Synchronizer` with `AutoMemory` as the counterpart component for cold-write operations
- Updated naming conventions in all related configuration files (e.g., `frontmatter:read` → `frontmatter_read`)
- Added new step definitions such as `submit_slug_updates` and `auto_memory`
- Updated relevant documentation
- Modified log output format for improved readability
* refactor(evolve): Refactor the auto-memory module and update related configurations
- Remove the old slug update commit step file
- Add new auto-memory planner and writer steps
- Update __init__.py to export the new step classes
- Modify the auto_memory configuration structure in default.yaml
- Update the slug field description for clearer explanation of its purpose
* up
* up
* refactor(tests): Move unit test directory from `tests4/unittest` to `tests4/unit`
Additionally, the assertion logic in test files has been updated: direct comparisons of `payload["notes"]` have been replaced with checks verifying the presence of paths and metadata within the response content. Furthermore, some test expectations have been simplified—for example, using `count` instead of asserting against specific note lists.
Specific changes include:
- Updating workflow configurations to align with the new test directory structure
- Modifying assertions across multiple test methods to make them more flexible and maintainable
- Cleaning up and optimizing parts of the test code structure
This is a comprehensive test refactoring effort aimed at improving test readability and robustness.
* Refactor(steps): Update memory writing logic and optimize JSON schema structure
Improved the write strategy description in `auto_memory_writer.yaml` to emphasize using `edit` over `write`.
Adjusted the `json_schema` structure in `base_step.py` to support the new function definition format.
Also corrected grammatical issues in the related documentation.
* Fix: Improve frontend data parsing error handling and update test files
Added capture and handling logic for YAML parsing exceptions, providing more detailed error messages when frontend data format issues occur. Also corrected the description text in a test file.
* fix(bm25_index): 修正BM25索引计算中的文档长度归一化问题
修复了在计算BM25相似度时对文档长度进行不正确归一化的bug，确保所有查询都能得到准确的相关性评分。
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* refactor(steps): Rename and adjust indexing step logic
- Rename `scan_changes.py` and `reindex.py` to `clear_and_scan.py`
- Update implementation details of `ScanChangesStep` and `ClearAndScanStep`
- Modify the scheduling mechanism in `WatchChangesStep`
- Adjust step registration and parameter configuration in config files
- Update related tests to align with the new interface changes
* up
* feat(daily): replace daily CRUD operations with slug provisioning approach
* refactor(tests): migrate CRUD step tests from HTTP server to direct LocalFileStore
* up
* up
* up
* up
---------
Co-authored-by: huangsen <huangsen.huang@alibaba-inc.com>
* feat(config): add comprehensive job definitions for vault operations
- Add utility jobs like version, search, traverse, list, read, stat
- Include file operations like move, delete, upload, download
- Add daily workspace management jobs: daily_list, daily_resolve, daily_reindex
- Update descriptions to reflect vault-based operations instead of working_dir
- Add proper section headers and documentation for each job category
refactor(steps): reorganize step modules and remove demo steps
- Move steps into categorized packages: common, crud, frontmatter, daily, jobs
- Remove demo steps (DemoEchoStep1, DemoEchoStep2, StreamDemoStep1, StreamDemoStep2)
- Add new steps: InitStep for vault initialization, TraverseStep for graph traversal
- Update __init__.py to auto-import all step modules
- Organize imports by functionality (common, CRUD operations, frontmatter, daily)
feat(vault): implement vault-centric file operations and configuration
- Change default config to use vault_dir instead of working_dir
- Add environment variable support for embedding configuration
- Implement file watcher with lite backend for daily/digest directories
- Update search step to use 'name' instead of 'title' from frontmatter
- Create ResourceEntry schema for tracking uploaded assets
docs(steps): add comprehensive documentation for all step categories
- Document file-I/O split by blast radius (crud vs frontmatter packages)
- Add detailed descriptions for each step category and functionality
- Explain the purpose and usage patterns for different types of file operations
- Provide clear parameter documentation for all new job configurations
* fix(config): correct vault directory path and remove unused job configurations
- Fix vault_dir from 'vaultd' to 'vault' in default configuration
- Remove deprecated traverse and list job configurations
- Remove unused tag tooling configurations
- Remove background watch_file job configuration
refactor(steps): remove unused jobs module import
- Comment out jobs module import in steps/__init__.py
- This removes unused synchronizer and digester step registrations
refactor(tests): update import path and add pylint directive
- Update ResourceEntry import from reme4.schema to reme4.schema.resource_meta
- Add pylint disable directive for unused argument in test datetime mocks
* efactor(steps): remove unused modules from __all__
- Remove "background" module from __all__ list
- Remove "jobs" module from __all__ list
- These modules were no longer being used in the steps package
* feat(config): update vault directory structure and remove file watcher
- Change vault_dir reference from ./vault to ./vault in CLI example
- Add daily_dir, digest_dir, and resource_dir configuration options
- Remove file_watcher component configuration as it's no longer needed
- Update comment to reflect correct module name (reme4vault)
refactor(steps): add background step and remove deprecated init step
- Import and register background step module
- Remove deprecated InitStep from common steps
- Update __all__ export list to include background step
refactor(reindex): improve reindex step to scan vault directly
- Update docstring to reflect vault scanning instead of watcher sync
- Replace file watcher stop/start logic with direct vault path walking
- Add support for suffix filtering during reindex operation
- Use index_changes job to process found files
refactor(wikilink_utils): enhance inbound source lookup with link scope
- Import LinkScopeEnum for proper type handling
- Update get_inlinks call to use ALL scope for virtual targets
- Improve documentation for reverse-index lookup behavior
test(refactor): clean up test suite removing deprecated functionality
- Remove test_init_job and test_demo_job unit tests
- Update help job assertion to check for literal command format
- Change test directory from .reme to vault in CRUD tests
- Remove init and demo job calls from integration test
BREAKING CHANGE: Removes file_watcher component and init step
* style(steps): fix import formatting in __init__.py
Add proper spacing in the background module import statement
to maintain consistent code style and readability.
* refactor(config): change default vault directory from vault to .reme
Default dev config now points vault_dir at ./.reme so `python -m
reme4 start` can be run from the repo root and exercise the full
atomic-tool surface against the seeded test data.
BREAKING CHANGE: The default vault directory has been changed from
'vault' to '.reme' in the configuration.
* docs(reme4_report): fix markdown formatting and remove extra content
* refactor(file_parser): delegate wikilink extraction to WikilinkHandler
* fix(search): handle empty query case gracefully
- Replace assertion with conditional check for empty query
- Set response success to false when query is empty
- Return error message instead of throwing assertion error
- Maintain existing validation for other parameters
* feat: rename working_dir to vault_dir and update documentation
- Rename working_dir to vault_dir across the application
- Update documentation to reflect vault_dir instead of working_dir
- Change FileFrontMatter title field to name field
- Update .gitignore to include vault directory
- Modify file path descriptions to reference vault instead of working_dir
- Update related configuration and property names accordingly
* refactor(steps): rename working_path to vault_path in CRUD operations
- Rename parameter from `working_path` to `vault_path` in `resolve_path` function
- Update all usages in append, edit, read, and write steps to use `self.vault_path`
- Update documentation comments to reflect the new parameter name
- Update docstring in read.py to mention `vault_dir` instead of `vault`
test(chunked_file_parser): update frontmatter field from title to name
- Change frontmatter field from `title` to `name` in test cases
- Update comment in background steps test to reference `vault_path` instead of `working_path`
* refactor(schema): remove unused ResourceEntry import
* feat(file_graph): add link scope filtering to get_inlinks/get_outlinks
* feat(file-store): add scope parameter to link methods
* feat(file_store): add FAISS-backed local file store implementation
- Introduce FaissLocalFileStore class with vector search capabilities using FAISS IndexFlatIP
- Implement FAISS index persistence with binary format and JSON id-map sidecar
- Add automatic index rebuilding when sidecar files are missing or corrupted
- Support tombstone mechanism for efficient deletion and compaction
- Register 'faiss' component type in the registry system
- Add faiss-cpu dependency requirement to pyproject.toml
- Update configuration schema to use simplified parameter structure
- Enhance search step to support parameter override from runtime context
- Add comprehensive unit tests for FAISS store functionality
- Implement fallback to parent methods for basic CRUD operations
* refactor(search): simplify parameter retrieval logic
- Removed _param method that checked context and kwargs
- Directly use self.kwargs.get for all parameter retrievals
- Maintained same default values for vector_weight, candidate_multiplier, expand_links, and max_links_per_direction
- Reduced code complexity by eliminating redundant context checking logic
- Add comprehensive Markdown kernel section covering Obsidian compatibility
- Include detailed explanation of YAML front matter and wikilink formats
- Document smart slicing mechanism using Markdown AST instead of fixed tokens
- Explain graph indexing with bidirectional links and multiple backends
- Restructure sections with proper numbering from 4 to 7
- Move Markdown kernel section to appear before self-evolution features
- Add detailed explanations of auto-memory, auto-dream, and auto-link processes
- Document three-way hybrid search with RRF fusion and progressive expansion
- Include engineering value explanations for keyword indexing in Chinese context
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* feat(config): add daily_dir configuration and background job logging
- Added daily_dir setting with default value 'memory' to config
- Implemented logging for background job startup events
- Enhanced component start logic to handle background backend type
- Updated default YAML configuration structure
* refactor(file_parser): replace _get_relative_path with to_vault_relative method
- Remove redundant working_dir property from base file parser
- Add to_vault_relative method to base component for path resolution
- Update bare_file_parser to use new to_vault_relative method
- Update default_file_parser to use new to_vault_relative method
- Update linked_file_parser to use new to_vault_relative method
- Make working_path absolute in base_component and steps
- Simplify index_changes step by removing redundant base variable
- Consolidate path relative logic in single shared method
* docs(reme4): update report with detailed architecture sections
- Add comprehensive Markdown kernel section covering Obsidian compatibility
- Include detailed explanation of YAML front matter and wikilink formats
- Document smart slicing mechanism using Markdown AST instead of fixed tokens
- Explain graph indexing with bidirectional links and multiple backends
- Restructure sections with proper numbering from 4 to 7
- Move Markdown kernel section to appear before self-evolution features
- Add detailed explanations of auto-memory, auto-dream, and auto-link processes
- Document three-way hybrid search with RRF fusion and progressive expansion
- Include engineering value explanations for keyword indexing in Chinese context
Update MCPClient to accept action parameter during method calls
instead of requiring it during instantiation, making it consistent
with the new client interface.
fix(reme): update client usage patterns
Update all client usages to pass action parameter during method
calls instead of during client instantiation.
* feat: add Neo4j file graph support and markdown parser with wikilink extraction
- Add Neo4jFileGraph implementation for property-graph storage with
virtual/real node handling and link management
- Introduce LinkedFileParser for markdown files with frontmatter,
wikilink graph extraction, and full-skeleton chunking
- Update pyproject.toml to include pyyaml, mistletoe, and neo4j
dependencies
- Modify .gitignore to exclude /vault and structure.md
- Change reme CLI entry point from reme_ai.main to remecli.reme
- Register new neo4j and md components in respective registries
* refactor(file-graph): add chunk_ids support to Neo4jFileGraph
Add chunk_ids field to File node properties in Neo4jFileGraph to
enable better content chunk tracking and management.
BREAKING CHANGE: File node schema now includes chunk_ids property
which may affect existing integrations.
feat(parser): implement wikilink resolution logic
Move path resolution logic from utils/path_resolver to
linked_file_parser module and enhance wikilink resolution with
folder-note rule support and improved error handling.
fix(tests): update test assertions and variable names
Update test cases to reflect changes in data structures and
variable naming conventions across various components.
chore(config): update package entry point reference
Change reme CLI entry point from remecli.reme:main to
reme_ai.reme:main in pyproject.toml.
refactor(utils): remove deprecated path_resolver module
Remove the old path_resolver utility module as its functionality
has been moved to linked_file_parser.
docs(file-graph): update Neo4jFileGraph documentation
Update class docstrings and comments to reflect new chunk_ids
property and other structural changes.
style(formatting): adjust code formatting and line breaks
Minor formatting improvements including line length optimization
and consistent spacing adjustments throughout the codebase.
* fix(pyproject.toml): correct entry point for reme command
Change the entry point from "reme_ai.reme:main" to "reme_ai.main:main"
to fix the module reference for the reme command in project scripts.
