Add mistletoe as a project dependency for enhanced markdown parsing
capabilities. Refactor the LinkedFileParser to use a proper AST-based
approach with MdNode tree structure, replacing the previous flat token
processing method. The new implementation provides better handling of
markdown elements including tables, code fences, lists, and headings,
with improved chunking logic that maintains document structure in
generated content segments.
The changes include:
- Add mistletoe dependency to pyproject.toml
- Implement proper AST node representation with MdNode class
- Create recursive chunking algorithm with TOC preservation
- Add support for frontmatter extraction with FileFrontMatter schema
- Optimize leaf node splitting with proper boundary detection
- Include part numbering for split content pieces
- Replace legacy line-based chunking with AST tree approach that builds
a complete document skeleton with content inlined under relevant
sections
- Add new chunking parameters: chunk_chars (default 2000) and embed_toc
(default True) to control content size and TOC inclusion
- Implement recursive chunking algorithm that respects structural
boundaries (code lines, table rows, list items) and prevents splits
inside blocks
- Introduce part markers [Part X/N] for oversized leaf blocks that
require splitting
- Add CLI tool for inspecting parsed chunks and edges with options for
preview and configuration
- Refactor edge extraction to use FileEdge.from_text instead of
parse_wikilinks for consistency
BREAKING CHANGE: Chunk format changes significantly with full TOC
skeleton wrapping content, affecting embedding models expecting
breadcrumb prefixes.
docs(protocol): add typed edge link protocol documentation
Add comprehensive documentation for the link protocol supporting
typed edges in body text. This includes specification for three
legal inline forms (bare wikilink, line-level Dataview,
inline-bracketed Dataview), predicate syntax rules, and the
machine-managed Relations section convention for organizing
discovered edges.
fix(memory): update path reference from vault_root to working_dir
Change the memory_create operation's path anchoring from
vault_root to working_dir to maintain consistency with the
current working directory configuration.
refactor(components): remove edge_extractor module and simplify parsing
Remove the edge_extractor component module entirely and
inline edge extraction logic directly into LinkedFileParser
using parse_wikilinks utility. This simplifies the architecture
by eliminating the separate edge extraction component and
delegating edge discovery to the maintainer's enrichment operations.
feat(parser): update parse method signature and simplify edge extraction
Modify LinkedFileParser to return (FileNode, list[FileChunk])
tuple instead of ParsedFile, remove dependency on BaseEdgeExtractor,
and implement direct wikilink parsing from body text only.
```
docs: add ReMe2 architecture design documentation
- Add comprehensive design document (reme2.md) detailing the
three-layer architecture (L1/L2/L3) for the vault system
- Document new protocols for folder notes and memory management
- Specify interface contracts for memory_* and vault_* tools
- Outline implementation phases from current state to target
refactor: fix typo in personal retriever class
- Correct spelling error: 'retri eved_nodes' -> 'retrieved_nodes'
in PersonalRetriever.result assignment
chore: update gitignore with vault-related patterns
- Add '/vault' to ignore vault directory
- Add '/reme-plugin' to ignore plugin files
- Add '/reme2/vault' to ignore new vault implementation
```
refactor(component): rename file_store to chunk_store and update interfaces
- Rename BaseFileStore to BaseChunkStore and update component type
- Replace file_store property with chunk_store in BaseStep
- Add file_graph property to access file metadata from FileWatcher
- Update all storage backends (Chroma, Local, SQLite) to use chunk-focused APIs
- Remove file metadata handling from chunk stores (moved to FileGraph)
- Update search methods to use ChunkFilter instead of SearchFilter
- Remove file_store imports and add chunk_store imports
```
- Add file_parser component with default implementation
- Introduce SearchFilter schema for path and tag filtering
- Implement filter functionality in BaseFileStore and LocalFileStore
- Update file watcher to use parser-based filtering instead of suffix filters
- Register new FILE_PARSER component enum
- Add test_data directory to gitignore
refactor: improve component imports and initialization
- Fix relative imports in application.py
- Add file_parser import to component init
- Initialize registry dict when component type doesn't exist
- Remove circular import in HttpService by using string annotation
- Update config yaml to use proper component names
refactor: enhance file watcher architecture
- Replace MdFileWatcher with more flexible FullFileWatcher and LightFileWatcher
- Remove suffix-based filtering in favor of parser-based approach
- Update BaseFileWatcher to resolve parsers from app context
- Remove unused watch_filter method
refactor: update ReMe core functionality
- Remove memory_path creation
- Simplify dream and proactive methods to return empty strings
- Update config defaults for HTTP service and component backends
docs: update component configuration in paw.yaml
- Change service backend from cmd to http
- Rename components to use correct singular forms
- Add default file parser and file watcher configurations
- Set up local file store with default settings
```
Co-authored-by: huangsen <huangsen.huang@alibaba-inc.com>
- Introduce BaseAsTokenCounter and EstimatedAsTokenCounter for token estimation
- Add AsMsgStat and AsBlockStat schema for message statistics tracking
- Implement FileIO class with read/write/append/edit operations
- Create file utility functions for safe async file reading and truncation
- Add MemorySearch component for semantic search in memory files
- Register new component types in ComponentEnum and update imports
- Add constants for default host, port, and truncation limits
- Create BaseService abstract base class for service implementations
- Implement BaseStep with component accessors and lifecycle management
- Add proper __all__ exports for all new modules and components
2026-04-16 20:21:04 +08:00
Renamed from reme_cli/schema/__init__.py (Browse further)
