* refactor(embedding): replace embedding model with embedding store architecture
- Remove as_token_counter component and its estimated token counter implementation
- Replace BaseEmbeddingModel with BaseEmbedding that wraps AgentScope embedding models
- Add support for multiple embedding providers (OpenAI, DashScope, Gemini, Ollama)
- Introduce BaseEmbeddingStore and LocalEmbeddingStore for caching and persistence
- Update component registry to use new embedding and embedding_store types
- Modify file stores to use embedding_store instead of embedding_model
- Update health check to monitor embedding_store instead of embedding_model
- Change default config to use embedding_store with local backend
- Add estimate_token_count utility function to utils module
* refactor(llm): replace as_llm components with unified llm implementation
- Remove deprecated as_llm and as_llm_formatter modules
- Add new llm module with BaseLLM and provider-specific implementations
- Update component registry to use LLM instead of AS_LLM
- Replace all as_llm/as_llm_formatter references with llm in steps
- Update configuration schema to use llm instead of as_llm
- Rename integration test file from test_as_llm to test_llm
- Add proper docstrings to embedding store dimension property
- Add pylint disable comment for embedding model call
- Remove unused FormatterBase import in base_step
- Update token_utils with function docstring
* refactor(evolve): replace ReActAgent with Agent and update message handling
- Removed FlexReActAgent class and direct ReActAgent imports
- Updated Agent instantiation to use new constructor parameters
- Changed message content to use TextBlock format instead of plain strings
- Modified timestamp access from msg.timestamp to msg.created_at
- Updated metadata access pattern for structured outputs
- Replaced Msg.from_dict with Msg.model_validate in auto_memory.py
- Updated test mocks to patch Agent instead of ReActAgent
- Changed message serialization from to_dict to model_dump in tests
- Moved component references to base class definition
- Updated demo tools to return strings instead of ToolResponse objects
* feat(step): migrate to FunctionTool and add streaming support
- Replace deprecated ToolResponse with FunctionTool in base_step.py
- Remove unused TextBlock import from base_step.py
- Update job registration to use new FunctionTool API
- Add thinking_budget parameter to llm_demo configuration
- Introduce StreamLLMDemoStep with streaming output capability
- Add structured output support to LLMDemoStep via generate_structured_output
- Implement streaming event handling for text/thinking/tool calls
- Add integration tests for embedding functionality
- Add integration tests for structured output and streaming features
- Update tool usage in demo steps to use new function naming convention
* fix(ci): correct package installation path in unittest workflow
- Updated pip install command to use proper package path "./reme4[dev,core]"
- Fixed dependency installation step in CI workflow configuration
* chore(workflow): update python versions in unittest workflow
- Remove Python 3.10 from test matrix
- Add Python 3.11 to test matrix
- Add Python 3.12 to test matrix
- Keep Python 3.13 in test matrix
- Update matrix configuration for better version coverage
* fix(health): handle missing dimensions attribute in embedding status
- Wrap dimensions access in try-except to prevent AttributeError
- Return None when dimensions attribute is not available
- Maintain backward compatibility for components without dimensions
test(component): add comprehensive tests for BaseComponent and related classes
- Add tests for Dependency class including repr and attribute access
- Add tests for bind method with various scenarios and edge cases
- Add tests for lifecycle management and async context handling
- Add tests for standalone and context-bound dependency resolution
- Add tests for ComponentMixin path utilities
test(common): update LocalFileStore initialization parameter
- Change embedding_model parameter to embedding_store in test setup
- Update all affected test files consistently
test(registry): add complete test suite for ComponentRegistry
- Add tests for register method with explicit names and defaults
- Add tests for decorator registration pattern
- Add tests for get_all method returning copies
- Add tests for unregister and clear operations
- Add tests for error handling of invalid registrations
test(job): add comprehensive tests for BaseJob and BackgroundJob
- Add tests for step resolution and exception handling
- Add tests for backoff delay calculation with jitter
- Add tests for supervisor loop restart behavior
- Add tests for task shutdown and cancellation
test(prompt): add complete test suite for PromptHandler
- Add tests for prompt loading from dictionaries and files
- Add tests for internationalization and language fallback
- Add tests for flag filtering and variable substitution
- Add tests for format validation and error handling
test(runtime): add basic tests for RuntimeContext dictionary access
- Add tests for item getting, setting and containment checks
- Add tests for missing key error handling
* feat(evolve): add permission context and agent state management
- Import PermissionContext, PermissionMode and AgentState modules
- Add state configuration with bypass permission mode to AutoDream agents
- Add state configuration with bypass permission mode to AutoMemory agents
- Implement static _to_msg method for message validation and formatting
- Refactor message processing to use the new _to_msg method
- Ensure proper content structure for text blocks in message conversion
* style(tests): update test files with linting rules and code improvements
- Add missing pylint disable directives for docstring and attribute warnings
- Replace lambda expressions with proper function definitions in test cases
- Import Path directly instead of using lambda with __import__
- Simplify assertion checks by using truthiness instead of equality to empty dict
- Remove unused imports and reorder imports consistently
- Format dictionary literals with proper indentation and line breaks
* refactor(components): extract shared component state into mixin
- Introduce ComponentMixin class with shared state for components and steps
- Move identity, config, and vault path functionality to ComponentMixin
- Update BaseComponent to inherit from ComponentMixin
- Update BaseStep to inherit from ComponentMixin
- Consolidate vault path helper methods in ComponentMixin
- Remove duplicate vault path implementations from BaseComponent and BaseStep
- Add ComponentMixin to components module exports
* refactor(file_io): implement path locks cache eviction mechanism
- Add _PATH_LOCKS_MAX constant set to 1024 for cache size limit
- Implement cache eviction logic when locks exceed maximum capacity
- Remove half of unlocked entries when cache limit is reached
- Use list comprehension to identify unlocked locks for removal
- Maintain existing path normalization and locking behavior
fix(edit): correct method call from public to private fail method
- Change self.fail to self._fail for internal error handling
- Maintain consistent private method usage within class
fix(mcp_client): change pop to get for optional command and args
- Replace kwargs.pop with kwargs.get to avoid removing keys
- Preserve original kwargs dictionary contents
- Maintain default empty string and list values
feat(reme): add client backend validation with error raising
- Check if client_cls is None before instantiation
- Raise ValueError with descriptive message for unknown backends
- Provide clear error feedback for invalid backend configurations
* fix(components): move directory creation to start method
- Moved component_metadata_path.mkdir call from __init__ to _start in base_keyword_index
- Moved component_metadata_path.mkdir call from __init__ to _start in local_file_graph
- Moved component_metadata_path.mkdir call from __init__ to _start in local_file_store
- Ensures directory creation happens after component initialization
- Prevents potential issues with path creation during object construction
* fix(steps): replace assertions with runtime errors for app_context validation
- Replace assert statements with explicit RuntimeError exceptions when app_context is None
- Add descriptive error messages for better debugging when resolving components
- Replace assert in resolve_component method with proper exception handling
- Replace assert in get_file_parser method with proper exception handling
- Maintain same functionality while improving error reporting clarity
* refactor(file_io): split file IO utilities into modular components
- Move daily note helpers to separate _daily_index module
- Extract path validation and resolution to new _path module
- Remove unused code and imports from _file_io module
- Update import statements across affected modules
- Introduce WikilinkHandler utility for link parsing
- Replace regex-based link extraction with WikilinkHandler
- Add integration JSONL files to gitignore
- Consolidate file locking mechanism in _file_io module
* style(formatter): fix spacing issues in file IO and chunked file parser
- Fixed whitespace around colon in slice notation in file_io.py
- Corrected spacing around colon in slice notation in chunked_file_parser.py
- Applied consistent formatting for array slicing operations
- Improved code readability by standardizing space placement in ranges
* refactor(steps): replace property-based component resolution with Ref descriptor
- Introduce Ref descriptor class for lazy component dependency resolution
- Replace _resolve method and individual properties with Ref descriptors
- Add as_llm, as_llm_formatter, as_token_counter, file_store, and embedding Ref attributes
- Remove legacy property methods and resolve logic from BaseStep
- Add cache clearing mechanism for Ref values during step calls
- Update UpdateCatalogStep to use Ref instead of property-based resolution
* refactor(evolve): consolidate auto memory planner and writer into single step
- Removed separate AutoMemoryPlannerStep and AutoMemoryWriterStep classes
- Combined functionality into new AutoMemoryStep class in auto_memory.py
- Migrated prompt templates from separate YAML files to unified auto_memory.yaml
- Updated module imports to reference new consolidated step
- Simplified memory recording process using single ReAct agent instead of two-stage planning/writing
- Maintained same input/output contract with messages, session_id, and memory_hint parameters
- Preserved all original functionality for creating/updating daily notes with conversation facts
* fix(daily): update empty session_id handling to create day-level file
- Changed test to verify empty session_id creates day-level file daily/<date>.md
- Updated assertion to check response success instead of rejection
- Modified metadata verification to include path, session_id and created status
- Added file existence check for the generated daily markdown file
- Updated test name and print statement to reflect new behavior
- Fixed test registration to use updated function name
- Rename slug parameter to session_id across daily note operations
- Update validation function from validate_slug to validate_session_id
- Change data structure keys from slug to session_id in note objects
- Modify file paths to use session_id instead of slug in daily folder
- Update documentation and comments to reflect session_id terminology
- Adjust test cases to use session_id parameter instead of slug
- Change default frontmatter to include empty description field
- Update configuration files to use session_id parameter name
- Modify scan_notes function to return session_id instead of slug
* refactor(steps): update naming conventions in components and configuration
Updated naming conventions across multiple files, changing colon-separated names to underscore-separated format, and added new step definitions along with documentation updates.
Key changes:
- Replaced `Synchronizer` with `AutoMemory` as the counterpart component for cold-write operations
- Updated naming conventions in all related configuration files (e.g., `frontmatter:read` → `frontmatter_read`)
- Added new step definitions such as `submit_slug_updates` and `auto_memory`
- Updated relevant documentation
- Modified log output format for improved readability
* refactor(evolve): Refactor the auto-memory module and update related configurations
- Remove the old slug update commit step file
- Add new auto-memory planner and writer steps
- Update __init__.py to export the new step classes
- Modify the auto_memory configuration structure in default.yaml
- Update the slug field description for clearer explanation of its purpose
* up
* up
* refactor(tests): Move unit test directory from `tests4/unittest` to `tests4/unit`
Additionally, the assertion logic in test files has been updated: direct comparisons of `payload["notes"]` have been replaced with checks verifying the presence of paths and metadata within the response content. Furthermore, some test expectations have been simplified—for example, using `count` instead of asserting against specific note lists.
Specific changes include:
- Updating workflow configurations to align with the new test directory structure
- Modifying assertions across multiple test methods to make them more flexible and maintainable
- Cleaning up and optimizing parts of the test code structure
This is a comprehensive test refactoring effort aimed at improving test readability and robustness.
* Refactor(steps): Update memory writing logic and optimize JSON schema structure
Improved the write strategy description in `auto_memory_writer.yaml` to emphasize using `edit` over `write`.
Adjusted the `json_schema` structure in `base_step.py` to support the new function definition format.
Also corrected grammatical issues in the related documentation.
* Fix: Improve frontend data parsing error handling and update test files
Added capture and handling logic for YAML parsing exceptions, providing more detailed error messages when frontend data format issues occur. Also corrected the description text in a test file.
* feat(file_io): port orthogonal crud_steps features onto upstream restructure
* refactor(file_io): expose with_neighbors/max_neighbors_per_direction/max_bytes as step kwargs (not LLM params)
Match search_step's convention: tuning knobs that are config-like (not part
of the LLM-facing schema) live in the yaml steps: block and are read via
self.kwargs.get(...) — not exposed under parameters.properties.
Also simplify the write step metadata field description.
* fix(bm25_index): 修正BM25索引计算中的文档长度归一化问题
修复了在计算BM25相似度时对文档长度进行不正确归一化的bug，确保所有查询都能得到准确的相关性评分。
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* refactor(steps): Rename and adjust indexing step logic
- Rename `scan_changes.py` and `reindex.py` to `clear_and_scan.py`
- Update implementation details of `ScanChangesStep` and `ClearAndScanStep`
- Modify the scheduling mechanism in `WatchChangesStep`
- Adjust step registration and parameter configuration in config files
- Update related tests to align with the new interface changes
* up
* feat(daily): replace daily CRUD operations with slug provisioning approach
* refactor(tests): migrate CRUD step tests from HTTP server to direct LocalFileStore
* up
* up
* up
* up
---------
Co-authored-by: huangsen <huangsen.huang@alibaba-inc.com>
* refactor(config): streamline job descriptions and parameter docs
- Simplify descriptions for search, traverse, list, read, stat,
frontmatter:read, write, edit, append, frontmatter:update,
frontmatter:delete, move, delete, upload, upload_resource, and
download jobs
- Shorten parameter descriptions to be more concise
- Maintain essential information while reducing verbosity
refactor(steps): rename daily steps and consolidate functionality
- Rename daily_resolve_step to daily_read_step
- Rename daily_create_step to daily_write_step
- Update __init__.py imports to reflect new step names
- Consolidate daily operations documentation
refactor(daily): extract helper functions and improve structure
- Rename _day_index.py to _daily_io.py
- Extract validate_slug function for Windows-safe filename validation
- Move scan_notes function to public interface
- Add comprehensive docstrings explaining slug validation and day-index
rebuild concerns
feat(daily): decouple list operation from index refresh
- Remove automatic day index refresh from daily_list_step
- Change daily_list_step to pure read operation with no side effects
- Sort notes by slug for stable output
- Update documentation to clarify read/write separation
refactor(daily): remove deprecated create step
- Remove unused daily/create.py module
- Simplify daily operations to focus on CRUD patterns
* refactor(daily): replace module imports with explicit step class imports
* feat(config): add comprehensive job definitions for vault operations
- Add utility jobs like version, search, traverse, list, read, stat
- Include file operations like move, delete, upload, download
- Add daily workspace management jobs: daily_list, daily_resolve, daily_reindex
- Update descriptions to reflect vault-based operations instead of working_dir
- Add proper section headers and documentation for each job category
refactor(steps): reorganize step modules and remove demo steps
- Move steps into categorized packages: common, crud, frontmatter, daily, jobs
- Remove demo steps (DemoEchoStep1, DemoEchoStep2, StreamDemoStep1, StreamDemoStep2)
- Add new steps: InitStep for vault initialization, TraverseStep for graph traversal
- Update __init__.py to auto-import all step modules
- Organize imports by functionality (common, CRUD operations, frontmatter, daily)
feat(vault): implement vault-centric file operations and configuration
- Change default config to use vault_dir instead of working_dir
- Add environment variable support for embedding configuration
- Implement file watcher with lite backend for daily/digest directories
- Update search step to use 'name' instead of 'title' from frontmatter
- Create ResourceEntry schema for tracking uploaded assets
docs(steps): add comprehensive documentation for all step categories
- Document file-I/O split by blast radius (crud vs frontmatter packages)
- Add detailed descriptions for each step category and functionality
- Explain the purpose and usage patterns for different types of file operations
- Provide clear parameter documentation for all new job configurations
* fix(config): correct vault directory path and remove unused job configurations
- Fix vault_dir from 'vaultd' to 'vault' in default configuration
- Remove deprecated traverse and list job configurations
- Remove unused tag tooling configurations
- Remove background watch_file job configuration
refactor(steps): remove unused jobs module import
- Comment out jobs module import in steps/__init__.py
- This removes unused synchronizer and digester step registrations
refactor(tests): update import path and add pylint directive
- Update ResourceEntry import from reme4.schema to reme4.schema.resource_meta
- Add pylint disable directive for unused argument in test datetime mocks
* efactor(steps): remove unused modules from __all__
- Remove "background" module from __all__ list
- Remove "jobs" module from __all__ list
- These modules were no longer being used in the steps package
* feat(config): update vault directory structure and remove file watcher
- Change vault_dir reference from ./vault to ./vault in CLI example
- Add daily_dir, digest_dir, and resource_dir configuration options
- Remove file_watcher component configuration as it's no longer needed
- Update comment to reflect correct module name (reme4vault)
refactor(steps): add background step and remove deprecated init step
- Import and register background step module
- Remove deprecated InitStep from common steps
- Update __all__ export list to include background step
refactor(reindex): improve reindex step to scan vault directly
- Update docstring to reflect vault scanning instead of watcher sync
- Replace file watcher stop/start logic with direct vault path walking
- Add support for suffix filtering during reindex operation
- Use index_changes job to process found files
refactor(wikilink_utils): enhance inbound source lookup with link scope
- Import LinkScopeEnum for proper type handling
- Update get_inlinks call to use ALL scope for virtual targets
- Improve documentation for reverse-index lookup behavior
test(refactor): clean up test suite removing deprecated functionality
- Remove test_init_job and test_demo_job unit tests
- Update help job assertion to check for literal command format
- Change test directory from .reme to vault in CRUD tests
- Remove init and demo job calls from integration test
BREAKING CHANGE: Removes file_watcher component and init step
* style(steps): fix import formatting in __init__.py
Add proper spacing in the background module import statement
to maintain consistent code style and readability.
* refactor(config): change default vault directory from vault to .reme
Default dev config now points vault_dir at ./.reme so `python -m
reme4 start` can be run from the repo root and exercise the full
atomic-tool surface against the seeded test data.
BREAKING CHANGE: The default vault directory has been changed from
'vault' to '.reme' in the configuration.
* docs(reme4_report): fix markdown formatting and remove extra content
* refactor(file_parser): delegate wikilink extraction to WikilinkHandler
* fix(search): handle empty query case gracefully
- Replace assertion with conditional check for empty query
- Set response success to false when query is empty
- Return error message instead of throwing assertion error
- Maintain existing validation for other parameters
* feat: rename working_dir to vault_dir and update documentation
- Rename working_dir to vault_dir across the application
- Update documentation to reflect vault_dir instead of working_dir
- Change FileFrontMatter title field to name field
- Update .gitignore to include vault directory
- Modify file path descriptions to reference vault instead of working_dir
- Update related configuration and property names accordingly
* refactor(steps): rename working_path to vault_path in CRUD operations
- Rename parameter from `working_path` to `vault_path` in `resolve_path` function
- Update all usages in append, edit, read, and write steps to use `self.vault_path`
- Update documentation comments to reflect the new parameter name
- Update docstring in read.py to mention `vault_dir` instead of `vault`
test(chunked_file_parser): update frontmatter field from title to name
- Change frontmatter field from `title` to `name` in test cases
- Update comment in background steps test to reference `vault_path` instead of `working_path`
* refactor(schema): remove unused ResourceEntry import
* feat(file_graph): add link scope filtering to get_inlinks/get_outlinks
* feat(file-store): add scope parameter to link methods
* feat(file_store): add FAISS-backed local file store implementation
- Introduce FaissLocalFileStore class with vector search capabilities using FAISS IndexFlatIP
- Implement FAISS index persistence with binary format and JSON id-map sidecar
- Add automatic index rebuilding when sidecar files are missing or corrupted
- Support tombstone mechanism for efficient deletion and compaction
- Register 'faiss' component type in the registry system
- Add faiss-cpu dependency requirement to pyproject.toml
- Update configuration schema to use simplified parameter structure
- Enhance search step to support parameter override from runtime context
- Add comprehensive unit tests for FAISS store functionality
- Implement fallback to parent methods for basic CRUD operations
* refactor(search): simplify parameter retrieval logic
- Removed _param method that checked context and kwargs
- Directly use self.kwargs.get for all parameter retrievals
- Maintained same default values for vector_weight, candidate_multiplier, expand_links, and max_links_per_direction
- Reduced code complexity by eliminating redundant context checking logic
- Add comprehensive Markdown kernel section covering Obsidian compatibility
- Include detailed explanation of YAML front matter and wikilink formats
- Document smart slicing mechanism using Markdown AST instead of fixed tokens
- Explain graph indexing with bidirectional links and multiple backends
- Restructure sections with proper numbering from 4 to 7
- Move Markdown kernel section to appear before self-evolution features
- Add detailed explanations of auto-memory, auto-dream, and auto-link processes
- Document three-way hybrid search with RRF fusion and progressive expansion
- Include engineering value explanations for keyword indexing in Chinese context
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* feat(config): add daily_dir configuration and background job logging
- Added daily_dir setting with default value 'memory' to config
- Implemented logging for background job startup events
- Enhanced component start logic to handle background backend type
- Updated default YAML configuration structure
* refactor(file_parser): replace _get_relative_path with to_vault_relative method
- Remove redundant working_dir property from base file parser
- Add to_vault_relative method to base component for path resolution
- Update bare_file_parser to use new to_vault_relative method
- Update default_file_parser to use new to_vault_relative method
- Update linked_file_parser to use new to_vault_relative method
- Make working_path absolute in base_component and steps
- Simplify index_changes step by removing redundant base variable
- Consolidate path relative logic in single shared method
* docs(reme4): update report with detailed architecture sections
- Add comprehensive Markdown kernel section covering Obsidian compatibility
- Include detailed explanation of YAML front matter and wikilink formats
- Document smart slicing mechanism using Markdown AST instead of fixed tokens
- Explain graph indexing with bidirectional links and multiple backends
- Restructure sections with proper numbering from 4 to 7
- Move Markdown kernel section to appear before self-evolution features
- Add detailed explanations of auto-memory, auto-dream, and auto-link processes
- Document three-way hybrid search with RRF fusion and progressive expansion
- Include engineering value explanations for keyword indexing in Chinese context
* feat: add Neo4j file graph support and markdown parser with wikilink extraction
- Add Neo4jFileGraph implementation for property-graph storage with
virtual/real node handling and link management
- Introduce LinkedFileParser for markdown files with frontmatter,
wikilink graph extraction, and full-skeleton chunking
- Update pyproject.toml to include pyyaml, mistletoe, and neo4j
dependencies
- Modify .gitignore to exclude /vault and structure.md
- Change reme CLI entry point from reme_ai.main to remecli.reme
- Register new neo4j and md components in respective registries
* refactor(file-graph): add chunk_ids support to Neo4jFileGraph
Add chunk_ids field to File node properties in Neo4jFileGraph to
enable better content chunk tracking and management.
BREAKING CHANGE: File node schema now includes chunk_ids property
which may affect existing integrations.
feat(parser): implement wikilink resolution logic
Move path resolution logic from utils/path_resolver to
linked_file_parser module and enhance wikilink resolution with
folder-note rule support and improved error handling.
fix(tests): update test assertions and variable names
Update test cases to reflect changes in data structures and
variable naming conventions across various components.
chore(config): update package entry point reference
Change reme CLI entry point from remecli.reme:main to
reme_ai.reme:main in pyproject.toml.
refactor(utils): remove deprecated path_resolver module
Remove the old path_resolver utility module as its functionality
has been moved to linked_file_parser.
docs(file-graph): update Neo4jFileGraph documentation
Update class docstrings and comments to reflect new chunk_ids
property and other structural changes.
style(formatting): adjust code formatting and line breaks
Minor formatting improvements including line length optimization
and consistent spacing adjustments throughout the codebase.
* fix(pyproject.toml): correct entry point for reme command
Change the entry point from "reme_ai.reme:main" to "reme_ai.main:main"
to fix the module reference for the reme command in project scripts.
