Add mistletoe as a project dependency for enhanced markdown parsing
capabilities. Refactor the LinkedFileParser to use a proper AST-based
approach with MdNode tree structure, replacing the previous flat token
processing method. The new implementation provides better handling of
markdown elements including tables, code fences, lists, and headings,
with improved chunking logic that maintains document structure in
generated content segments.
The changes include:
- Add mistletoe dependency to pyproject.toml
- Implement proper AST node representation with MdNode class
- Create recursive chunking algorithm with TOC preservation
- Add support for frontmatter extraction with FileFrontMatter schema
- Optimize leaf node splitting with proper boundary detection
- Include part numbering for split content pieces
- Replace legacy line-based chunking with AST tree approach that builds
a complete document skeleton with content inlined under relevant
sections
- Add new chunking parameters: chunk_chars (default 2000) and embed_toc
(default True) to control content size and TOC inclusion
- Implement recursive chunking algorithm that respects structural
boundaries (code lines, table rows, list items) and prevents splits
inside blocks
- Introduce part markers [Part X/N] for oversized leaf blocks that
require splitting
- Add CLI tool for inspecting parsed chunks and edges with options for
preview and configuration
- Refactor edge extraction to use FileEdge.from_text instead of
parse_wikilinks for consistency
BREAKING CHANGE: Chunk format changes significantly with full TOC
skeleton wrapping content, affecting embedding models expecting
breadcrumb prefixes.
docs(protocol): add typed edge link protocol documentation
Add comprehensive documentation for the link protocol supporting
typed edges in body text. This includes specification for three
legal inline forms (bare wikilink, line-level Dataview,
inline-bracketed Dataview), predicate syntax rules, and the
machine-managed Relations section convention for organizing
discovered edges.
fix(memory): update path reference from vault_root to working_dir
Change the memory_create operation's path anchoring from
vault_root to working_dir to maintain consistency with the
current working directory configuration.
refactor(components): remove edge_extractor module and simplify parsing
Remove the edge_extractor component module entirely and
inline edge extraction logic directly into LinkedFileParser
using parse_wikilinks utility. This simplifies the architecture
by eliminating the separate edge extraction component and
delegating edge discovery to the maintainer's enrichment operations.
feat(parser): update parse method signature and simplify edge extraction
Modify LinkedFileParser to return (FileNode, list[FileChunk])
tuple instead of ParsedFile, remove dependency on BaseEdgeExtractor,
and implement direct wikilink parsing from body text only.
```
feat(file_watcher): add directory deletion support with descendant indexing
Add support for deleting entire directories and their indexed descendants
in the file watcher. Previously only individual file deletions were
handled properly. Now when a directory is deleted, the system finds all
indexed files beneath that directory path and removes them along with
their metadata and chunks.
The implementation includes:
- New `_descendant_indexed_paths` method to find all indexed files
under a given directory path
- Updated `_on_deleted` method to process both the target path and
all its indexed descendants
- Proper handling of symlinks and path resolution differences
- Enhanced logging to show directory deletion with child count
Also adds necessary os import for path operations.
refactor(config): restructure configuration profiles for clarity
Rename curated.yaml to remove outdated configuration file and
rename full.yaml to expert.yaml with updated documentation.
Add new service.yaml configuration profile that provides a
service-aligned MCP surface with three main tools:
- retrieve: graph-aware hybrid retrieval
- remember: single write entry point with log/distill modes
- maintain: vault hygiene sweep
The expert configuration now excludes the ingest tool since
cold-path operations are handled by external agents, and adds
memory_lint tool for structural issue detection.
Updated documentation to clarify the different configuration
profiles and their intended usage patterns.
```
docs: add ReMe2 architecture design documentation
- Add comprehensive design document (reme2.md) detailing the
three-layer architecture (L1/L2/L3) for the vault system
- Document new protocols for folder notes and memory management
- Specify interface contracts for memory_* and vault_* tools
- Outline implementation phases from current state to target
refactor: fix typo in personal retriever class
- Correct spelling error: 'retri eved_nodes' -> 'retrieved_nodes'
in PersonalRetriever.result assignment
chore: update gitignore with vault-related patterns
- Add '/vault' to ignore vault directory
- Add '/reme-plugin' to ignore plugin files
- Add '/reme2/vault' to ignore new vault implementation
```
refactor(component): rename file_store to chunk_store and update interfaces
- Rename BaseFileStore to BaseChunkStore and update component type
- Replace file_store property with chunk_store in BaseStep
- Add file_graph property to access file metadata from FileWatcher
- Update all storage backends (Chroma, Local, SQLite) to use chunk-focused APIs
- Remove file metadata handling from chunk stores (moved to FileGraph)
- Update search methods to use ChunkFilter instead of SearchFilter
- Remove file_store imports and add chunk_store imports
```
- Removed hash, size, content, and chunk_count fields from FileMetadata model
- Updated chroma and local file stores to remove excluded fields from model dump
- Removed hash calculation and content storage from default file parser
- Removed chunk count tracking from file parsing logic
- Simplified FileMetadata schema to only include essential fields
- Added new SQLite file store implementation with vector and full-text search
- Added memory search step for semantic search functionality
- Remove abstract methods from base component start/close
- Update BaseJob to remove name parameter and simplify initialization
- Change file modification time field from mtime_ms to modified_time in seconds
- Add type checking imports and improve typing annotations
- Implement LocalFileStore with JSONL persistence for file chunks
- Add MdFileParser with markdown and frontmatter support
- Simplify HttpClient call method with proper kwargs handling
- Remove unused ReMe class methods and create backup version
- Update StreamJob to use step_components instead of steps attribute
- Introduce Application class for managing application lifecycle
- Add base component classes for LLM formatters and token counters
- Implement embedding model base with caching and batching support
- Create file watcher base with watchfiles integration
- Add job and step base components for workflow execution
- Update base component with async locks and improved lifecycle management
- Register new component types in component registry
- Add application context and runtime context for dependency injection
- Add file_parser component with default implementation
- Introduce SearchFilter schema for path and tag filtering
- Implement filter functionality in BaseFileStore and LocalFileStore
- Update file watcher to use parser-based filtering instead of suffix filters
- Register new FILE_PARSER component enum
- Add test_data directory to gitignore
refactor: improve component imports and initialization
- Fix relative imports in application.py
- Add file_parser import to component init
- Initialize registry dict when component type doesn't exist
- Remove circular import in HttpService by using string annotation
- Update config yaml to use proper component names
refactor: enhance file watcher architecture
- Replace MdFileWatcher with more flexible FullFileWatcher and LightFileWatcher
- Remove suffix-based filtering in favor of parser-based approach
- Update BaseFileWatcher to resolve parsers from app context
- Remove unused watch_filter method
refactor: update ReMe core functionality
- Remove memory_path creation
- Simplify dream and proactive methods to return empty strings
- Update config defaults for HTTP service and component backends
docs: update component configuration in paw.yaml
- Change service backend from cmd to http
- Rename components to use correct singular forms
- Add default file parser and file watcher configurations
- Set up local file store with default settings
```
Co-authored-by: huangsen <huangsen.huang@alibaba-inc.com>
- Introduce BaseAsTokenCounter and EstimatedAsTokenCounter for token estimation
- Add AsMsgStat and AsBlockStat schema for message statistics tracking
- Implement FileIO class with read/write/append/edit operations
- Create file utility functions for safe async file reading and truncation
- Add MemorySearch component for semantic search in memory files
- Register new component types in ComponentEnum and update imports
- Add constants for default host, port, and truncation limits
- Create BaseService abstract base class for service implementations
- Implement BaseStep with component accessors and lifecycle management
- Add proper __all__ exports for all new modules and components
