Add mistletoe as a project dependency for enhanced markdown parsing
capabilities. Refactor the LinkedFileParser to use a proper AST-based
approach with MdNode tree structure, replacing the previous flat token
processing method. The new implementation provides better handling of
markdown elements including tables, code fences, lists, and headings,
with improved chunking logic that maintains document structure in
generated content segments.
The changes include:
- Add mistletoe dependency to pyproject.toml
- Implement proper AST node representation with MdNode class
- Create recursive chunking algorithm with TOC preservation
- Add support for frontmatter extraction with FileFrontMatter schema
- Optimize leaf node splitting with proper boundary detection
- Include part numbering for split content pieces
- Replace legacy line-based chunking with AST tree approach that builds
a complete document skeleton with content inlined under relevant
sections
- Add new chunking parameters: chunk_chars (default 2000) and embed_toc
(default True) to control content size and TOC inclusion
- Implement recursive chunking algorithm that respects structural
boundaries (code lines, table rows, list items) and prevents splits
inside blocks
- Introduce part markers [Part X/N] for oversized leaf blocks that
require splitting
- Add CLI tool for inspecting parsed chunks and edges with options for
preview and configuration
- Refactor edge extraction to use FileEdge.from_text instead of
parse_wikilinks for consistency
BREAKING CHANGE: Chunk format changes significantly with full TOC
skeleton wrapping content, affecting embedding models expecting
breadcrumb prefixes.
Remove SqliteFileStore backend and simplify the base file store
interface. The file store now focuses on in-memory + JSONL persistence
for the (file → chunks) graph with reduced complexity.
BREAKING CHANGE: SqliteFileStore is removed, only LocalFileStore remains
available.
docs(protocol): add typed edge link protocol documentation
Add comprehensive documentation for the link protocol supporting
typed edges in body text. This includes specification for three
legal inline forms (bare wikilink, line-level Dataview,
inline-bracketed Dataview), predicate syntax rules, and the
machine-managed Relations section convention for organizing
discovered edges.
fix(memory): update path reference from vault_root to working_dir
Change the memory_create operation's path anchoring from
vault_root to working_dir to maintain consistency with the
current working directory configuration.
refactor(components): remove edge_extractor module and simplify parsing
Remove the edge_extractor component module entirely and
inline edge extraction logic directly into LinkedFileParser
using parse_wikilinks utility. This simplifies the architecture
by eliminating the separate edge extraction component and
delegating edge discovery to the maintainer's enrichment operations.
feat(parser): update parse method signature and simplify edge extraction
Modify LinkedFileParser to return (FileNode, list[FileChunk])
tuple instead of ParsedFile, remove dependency on BaseEdgeExtractor,
and implement direct wikilink parsing from body text only.
```
feat: add Claude Code marketplace plugin with service and expert tiers
Add comprehensive Claude Code marketplace plugin supporting two paradigms
for managing markdown vaults:
- reme-service: Service-tier plugin with high-level MCP tools
(retrieve/remember/maintain) where reme2 internals handle R-M-W loop
- reme-expert: Expert-tier plugin where Claude Code agent runs R-M-W loop
directly using raw memory_* primitives guided by reme protocol skill
Both plugins implement identical 4-phase work paradigm:
- Recall: Retrieve relevant context with ranking by relevance/proximity
- Log: Record event facts and raw materials via idempotent event-folder upsert
- Distill: Promote events to topic graph via R-M-W loop
- Maintain: Vault hygiene sweep with lint and decay operations
Include marketplace configuration, documentation, MCP server setup,
subagents (reme-distiller, reme-curator), hooks (PreCompact, SessionEnd,
Stop), and slash commands (/reme-distill, /reme-recall, /reme-clean).
Remove outdated plugin entries from gitignore.
```
feat(file_watcher): add directory deletion support with descendant indexing
Add support for deleting entire directories and their indexed descendants
in the file watcher. Previously only individual file deletions were
handled properly. Now when a directory is deleted, the system finds all
indexed files beneath that directory path and removes them along with
their metadata and chunks.
The implementation includes:
- New `_descendant_indexed_paths` method to find all indexed files
under a given directory path
- Updated `_on_deleted` method to process both the target path and
all its indexed descendants
- Proper handling of symlinks and path resolution differences
- Enhanced logging to show directory deletion with child count
Also adds necessary os import for path operations.
refactor(config): restructure configuration profiles for clarity
Rename curated.yaml to remove outdated configuration file and
rename full.yaml to expert.yaml with updated documentation.
Add new service.yaml configuration profile that provides a
service-aligned MCP surface with three main tools:
- retrieve: graph-aware hybrid retrieval
- remember: single write entry point with log/distill modes
- maintain: vault hygiene sweep
The expert configuration now excludes the ingest tool since
cold-path operations are handled by external agents, and adds
memory_lint tool for structural issue detection.
Updated documentation to clarify the different configuration
profiles and their intended usage patterns.
```
docs: add ReMe2 architecture design documentation
- Add comprehensive design document (reme2.md) detailing the
three-layer architecture (L1/L2/L3) for the vault system
- Document new protocols for folder notes and memory management
- Specify interface contracts for memory_* and vault_* tools
- Outline implementation phases from current state to target
refactor: fix typo in personal retriever class
- Correct spelling error: 'retri eved_nodes' -> 'retrieved_nodes'
in PersonalRetriever.result assignment
chore: update gitignore with vault-related patterns
- Add '/vault' to ignore vault directory
- Add '/reme-plugin' to ignore plugin files
- Add '/reme2/vault' to ignore new vault implementation
```
refactor(component): rename file_store to chunk_store and update interfaces
- Rename BaseFileStore to BaseChunkStore and update component type
- Replace file_store property with chunk_store in BaseStep
- Add file_graph property to access file metadata from FileWatcher
- Update all storage backends (Chroma, Local, SQLite) to use chunk-focused APIs
- Remove file metadata handling from chunk stores (moved to FileGraph)
- Update search methods to use ChunkFilter instead of SearchFilter
- Remove file_store imports and add chunk_store imports
```
- Create MCPService class to expose jobs as MCP tools
- Implement FastMCP integration with async lifespan management
- Add environment variable setup for service information
- Implement tool registration for job execution via FunctionTool
- Support multiple transport types (sse, stdio) with configurable host/port
- Integrate with application lifecycle for proper startup/shutdown
- replace watch_paths parameter with single watch_path string
- add FileGraph integration for tracking file relationships
- implement persistent graph storage in .reme directory
- add load/build graph functionality on start
- refactor file change handling with separate methods for add/modify/delete
- update scan_existing_files to work with single path
- add file filter to exclude meta directory from watching
- remove abstract method requirement and implement concrete change handlers
BREAKING CHANGE: watch_paths parameter changed to watch_path (single path)
- Introduce BaseFileParser abstract class with component registration
- Add MdFileParser implementation for markdown files with YAML frontmatter
- Create TextFileParser implementation with built-in chunking support
- Implement file suffix enumeration for parser type safety
- Add chunking logic with configurable token size and overlap
- Support text file parsing with error handling for encoding issues
- Include line number tracking and content hashing for file chunks
- Removed hash, size, content, and chunk_count fields from FileMetadata model
- Updated chroma and local file stores to remove excluded fields from model dump
- Removed hash calculation and content storage from default file parser
- Removed chunk count tracking from file parsing logic
- Simplified FileMetadata schema to only include essential fields
- Added new SQLite file store implementation with vector and full-text search
- Added memory search step for semantic search functionality
- Remove abstract methods from base component start/close
- Update BaseJob to remove name parameter and simplify initialization
- Change file modification time field from mtime_ms to modified_time in seconds
- Add type checking imports and improve typing annotations
- Implement LocalFileStore with JSONL persistence for file chunks
- Add MdFileParser with markdown and frontmatter support
- Simplify HttpClient call method with proper kwargs handling
- Remove unused ReMe class methods and create backup version
- Update StreamJob to use step_components instead of steps attribute
- Introduce Application class for managing application lifecycle
- Add base component classes for LLM formatters and token counters
- Implement embedding model base with caching and batching support
- Create file watcher base with watchfiles integration
- Add job and step base components for workflow execution
- Update base component with async locks and improved lifecycle management
- Register new component types in component registry
- Add application context and runtime context for dependency injection
- Integrate AnthropicChatModel with new AnthropicAsLLM component
- Add component formatters for OpenAI and Anthropic chat models
- Implement token counter component with estimated token counting
- Create base client component for ReMe service communication
- Refactor BaseComponent to remove app_context parameter from _start/_close
- Update embedding model base class to remove retry logic and use npz cache
- Add job component for sequential step execution with BaseJob
- Implement step component base class for LLM workflow execution
- Enhance application context with proper type annotations
- Update component initialization to pass app_context automatically
- Remove asyncio dependency from embedding model cache operations
feat(file-store): add Chroma and SQLite file store implementations
- Add ChromaFileStore and SqliteFileStore classes to support additional
storage backends
- Export new store classes in __init__.py module
refactor(file-store): enhance BaseFileStore with hybrid search capabilities
- Add keyword scoring utility with word-match ratio and phrase bonus
- Implement hybrid search method that combines vector and keyword results
- Add merge logic for combining vector and keyword search results with
weighted scoring
- Move abstract methods to separate section for better organization
- Remove redundant search filter parameter from local implementation
refactor(file-store): simplify LocalFileStore implementation
- Remove unused delete_file_chunks and upsert_chunks methods
- Remove redundant update_file_metadata method
- Update chunk counting logic to use file_meta directly
- Simplify keyword search to use new base class scoring utility
- Remove duplicate hybrid search implementation since it's now in base class
```
- Add file_parser component with default implementation
- Introduce SearchFilter schema for path and tag filtering
- Implement filter functionality in BaseFileStore and LocalFileStore
- Update file watcher to use parser-based filtering instead of suffix filters
- Register new FILE_PARSER component enum
- Add test_data directory to gitignore
refactor: improve component imports and initialization
- Fix relative imports in application.py
- Add file_parser import to component init
- Initialize registry dict when component type doesn't exist
- Remove circular import in HttpService by using string annotation
- Update config yaml to use proper component names
refactor: enhance file watcher architecture
- Replace MdFileWatcher with more flexible FullFileWatcher and LightFileWatcher
- Remove suffix-based filtering in favor of parser-based approach
- Update BaseFileWatcher to resolve parsers from app context
- Remove unused watch_filter method
refactor: update ReMe core functionality
- Remove memory_path creation
- Simplify dream and proactive methods to return empty strings
- Update config defaults for HTTP service and component backends
docs: update component configuration in paw.yaml
- Change service backend from cmd to http
- Rename components to use correct singular forms
- Add default file parser and file watcher configurations
- Set up local file store with default settings
```
Co-authored-by: huangsen <huangsen.huang@alibaba-inc.com>
- Changed condition from divisor equals zero to divisor less than or equal to zero
- Prevents division by zero error when estimate_divisor is negative
- Maintains validation logic for invalid divisor values
- Introduce BaseAsTokenCounter and EstimatedAsTokenCounter for token estimation
- Add AsMsgStat and AsBlockStat schema for message statistics tracking
- Implement FileIO class with read/write/append/edit operations
- Create file utility functions for safe async file reading and truncation
- Add MemorySearch component for semantic search in memory files
- Register new component types in ComponentEnum and update imports
- Add constants for default host, port, and truncation limits
- Create BaseService abstract base class for service implementations
- Implement BaseStep with component accessors and lifecycle management
- Add proper __all__ exports for all new modules and components
- Introduce BaseAsTokenCounter and EstimatedAsTokenCounter for token estimation
- Add AsMsgStat and AsBlockStat schema for message statistics tracking
- Implement FileIO class with read/write/append/edit operations
- Create file utility functions for safe async file reading and truncation
- Add MemorySearch component for semantic search in memory files
- Register new component types in ComponentEnum and update imports
- Add constants for default host, port, and truncation limits
- Create BaseService abstract base class for service implementations
- Implement BaseStep with component accessors and lifecycle management
- Add proper __all__ exports for all new modules and components
- Removed unused-argument pylint disable configuration
- Cleaned up tool.pylint.messages_control section from pyproject.toml
- Updated test configuration to use explicit ignore patterns instead of pylint disables
- Add BaseClient, BaseFileStore, BaseFileWatcher, BaseJob, BaseService, and BaseStep classes
- Move component initialization logic from ApplicationContext to Application class
- Add logo printing and logging initialization in Application startup
- Create client module with base client implementation
- Add file store base class with embedding resolution and validation
- Implement file watcher base class with watchfiles integration
- Add job base class for sequential step execution orchestration
- Create service base class for job exposure mechanisms
- Refactor BaseStep with LLM workflow execution capabilities
- Add case converter utility for naming convention transformations
- Update import structure and module organization
- Add proper type hints and docstrings across all components
- Implement component registry integration for dynamic loading
- Add error handling for missing backend configurations
- Replace ReMeClient with modular client implementations
- Add component registry with type-based registration system
- Introduce BaseClient extending BaseComponent with lifecycle management
- Create HttpClient with environment-based service discovery
- Add constants for default host/port configurations
- Update service info propagation through environment variables
- Restructure imports and exports across component modules
- Add run_coro_safely utility for safe coroutine execution
- Implement component type enumeration for better organization
- Register components with R decorator for automatic discovery
- Add placeholder methods for ReMe core functionalities
- Update command-line entry point to use dynamic client selection
- Implement BaseComponent with async lifecycle and context management
- Add ApplicationContext for managing component initialization and registry
- Create Application class for orchestrating job execution and lifecycle
- Add AS LLM components with OpenAI chat model wrapper
- Implement AS LLM formatter components with OpenAI formatter
- Add client implementations including base, HTTP and ReMe clients
- Create embedding model base class with caching and batching support
- Implement file store base class with vector and full-text search
- Add file watcher components for monitoring file system changes
- Create job components for executing workflows
- Implement service components for exposing jobs via different protocols
- Add configuration schema with ApplicationConfig and ComponentConfig
- Include utility modules for case conversion, chunking, logging and similarity
- Register component types and create component registry system
* fix(core): handle chromadb import error gracefully
- Changed CHROMADB_AVAILABLE flag to _CHROMADB_IMPORT_ERROR exception storage
- Updated version from 0.3.1.7 to 0.3.1.8
- Modified import error handling to preserve original exception details
- Removed hardcoded ImportError message in favor of dynamic exception raising
- Added proper logger initialization using get_logger utility
* refactor(file_store): move sqlite3 imports inside initialization methods
- Moved sqlite3 import from module level to inside init methods
- Removed unused import statement at top of file
- Maintains same functionality while improving import organization
- Prevents potential issues with early sqlite3 dependency loading
* refactor(core): update import error handling with broader exception types
- Changed ImportError to Exception for ray import error handling
- Updated chromadb import error to use Exception instead of ImportError
- Modified elasticsearch import error to catch general exceptions
- Changed asyncpg import error handling from ImportError to Exception
- Updated qdrant import error to use Exception instead of ImportError
- Added explicit type hints for all import error variables as Exception | None
- Changed CHROMADB_AVAILABLE flag to _CHROMADB_IMPORT_ERROR exception storage
- Updated version from 0.3.1.7 to 0.3.1.8
- Modified import error handling to preserve original exception details
- Removed hardcoded ImportError message in favor of dynamic exception raising
- Added proper logger initialization using get_logger utility
* feat(compactor): add extra instruction support and improve error handling
- Add extra_instruction parameter to compactor for custom guidance during message compaction
- Implement try-catch blocks around AS LLM initialization with detailed error logging
- Add extra_instruction parameter to ReMe.compact method with comprehensive documentation
- Update agentscope dependency from 1.0.17 to 1.0.18 in light installation
- Bump version number from 0.3.1.6 to 0.3.1.7
- Pass extra_instruction parameter through compactor instantiation and execution flow
* fix(core): add error handling for AS LLM formatters and token counters initialization
- Wrapped AS LLM formatters initialization in try-except blocks
- Added specific error logging for failed AS LLM formatter initialization
- Wrapped AS token counters initialization in try-except blocks
- Added specific error logging for failed AS token counter initialization
- Applied same error handling pattern to both initial setup and restart operations
- Maintained existing warning logs for unsupported backends
