Add mistletoe as a project dependency for enhanced markdown parsing
capabilities. Refactor the LinkedFileParser to use a proper AST-based
approach with MdNode tree structure, replacing the previous flat token
processing method. The new implementation provides better handling of
markdown elements including tables, code fences, lists, and headings,
with improved chunking logic that maintains document structure in
generated content segments.
The changes include:
- Add mistletoe dependency to pyproject.toml
- Implement proper AST node representation with MdNode class
- Create recursive chunking algorithm with TOC preservation
- Add support for frontmatter extraction with FileFrontMatter schema
- Optimize leaf node splitting with proper boundary detection
- Include part numbering for split content pieces
- Replace legacy line-based chunking with AST tree approach that builds
a complete document skeleton with content inlined under relevant
sections
- Add new chunking parameters: chunk_chars (default 2000) and embed_toc
(default True) to control content size and TOC inclusion
- Implement recursive chunking algorithm that respects structural
boundaries (code lines, table rows, list items) and prevents splits
inside blocks
- Introduce part markers [Part X/N] for oversized leaf blocks that
require splitting
- Add CLI tool for inspecting parsed chunks and edges with options for
preview and configuration
- Refactor edge extraction to use FileEdge.from_text instead of
parse_wikilinks for consistency
BREAKING CHANGE: Chunk format changes significantly with full TOC
skeleton wrapping content, affecting embedding models expecting
breadcrumb prefixes.
docs(protocol): add typed edge link protocol documentation
Add comprehensive documentation for the link protocol supporting
typed edges in body text. This includes specification for three
legal inline forms (bare wikilink, line-level Dataview,
inline-bracketed Dataview), predicate syntax rules, and the
machine-managed Relations section convention for organizing
discovered edges.
fix(memory): update path reference from vault_root to working_dir
Change the memory_create operation's path anchoring from
vault_root to working_dir to maintain consistency with the
current working directory configuration.
refactor(components): remove edge_extractor module and simplify parsing
Remove the edge_extractor component module entirely and
inline edge extraction logic directly into LinkedFileParser
using parse_wikilinks utility. This simplifies the architecture
by eliminating the separate edge extraction component and
delegating edge discovery to the maintainer's enrichment operations.
feat(parser): update parse method signature and simplify edge extraction
Modify LinkedFileParser to return (FileNode, list[FileChunk])
tuple instead of ParsedFile, remove dependency on BaseEdgeExtractor,
and implement direct wikilink parsing from body text only.
```
- Add file_parser component with default implementation
- Introduce SearchFilter schema for path and tag filtering
- Implement filter functionality in BaseFileStore and LocalFileStore
- Update file watcher to use parser-based filtering instead of suffix filters
- Register new FILE_PARSER component enum
- Add test_data directory to gitignore
refactor: improve component imports and initialization
- Fix relative imports in application.py
- Add file_parser import to component init
- Initialize registry dict when component type doesn't exist
- Remove circular import in HttpService by using string annotation
- Update config yaml to use proper component names
refactor: enhance file watcher architecture
- Replace MdFileWatcher with more flexible FullFileWatcher and LightFileWatcher
- Remove suffix-based filtering in favor of parser-based approach
- Update BaseFileWatcher to resolve parsers from app context
- Remove unused watch_filter method
refactor: update ReMe core functionality
- Remove memory_path creation
- Simplify dream and proactive methods to return empty strings
- Update config defaults for HTTP service and component backends
docs: update component configuration in paw.yaml
- Change service backend from cmd to http
- Rename components to use correct singular forms
- Add default file parser and file watcher configurations
- Set up local file store with default settings
```
Co-authored-by: huangsen <huangsen.huang@alibaba-inc.com>
* refactor(file_store): simplify ChromaDB client initialization and improve file truncation logic
- Remove shutil import and _create_chroma_client method from chroma_file_store.py
- Directly initialize ChromaDB PersistentClient in start method without retry logic
- Reduce DEFAULT_MAX_BYTES from 100KB to 50KB in file_utils.py
- Update truncation notice format to provide clearer continuation instructions
- Add _truncate_fresh and _retruncate functions for better text truncation handling
- Replace inline truncation logic with dedicated function calls in file_utils.py
- Rename skills_tool_ids to md_file_tool_ids in tool_result_compactor.py
- Update file detection logic to identify any .md files instead of only skill.md
- Create comprehensive unit tests for truncation functionality in test_truncate_text_output.py
* chore(version): bump version to 0.3.1.6
- Update __version__ from 0.3.1.5 to 0.3.1.6 in __init__.py
* refactor(core): replace text truncation utilities with new marker system
- Remove old truncate_text_utils module and its exports
- Replace TRUNCATION_MARKER_START with _TRUNCATION_NOTICE_MARKER constant
- Update as_msg_stat.py to split content using new marker format
- Modify FileIO tool to use TRUNCATION_NOTICE_MARKER for continuation hints
- Change is_truncated function checks to use marker presence detection
- Move transformers dependency from main deps to light extra dependencies
- Update tool result compactor tests to verify marker instead of is_truncated calls
* feat(file_io): enhance file operations with path resolution and append functionality
- Add expanduser() to resolve file paths with ~ symbol
- Implement proper file existence and type validation in update_file
- Add new append_file method to append content to files
- Update truncation notice format for better readability
- Fix typo in error message from "provide" to "provided"
- Update transformers dependency in pyproject.toml
- Remove duplicate transformers dependency from light extras
* refactor(file_io): disable pylint too-many-return-statements warning
* perf(file_watcher): increase default polling delay and optimize watcher configuration
- Increased default poll_delay_ms from 1000ms to 2000ms to reduce CPU usage
- Removed force_polling parameter as it's no longer needed with updated polling strategy
- Simplified async watch configuration by removing conditional force_polling logic
- Reduced overall system resource consumption during file watching operations
* refactor(memory): update conversation log documentation in memory summary
- Changed "Raw conversation logs" to "Earlier conversation logs" for clarity
- Added warning note about potentially large dialog file sizes
- Improved formatting with additional line break for better readability
- Maintained existing compressed summary integration unchanged
* feat(memory): add long-term memory support to file-based memory system
- Initialize _long_term_memory attribute as empty string
- Add memories section to content when long-term memory exists
- Consolidate summary and memories into single user message
- Format memories with markdown header # Memories
- Maintain existing compressed summary functionality
- Join multiple content parts with double newlines
* style(memory): update message formatting and improve logging
- Change default include_thinking parameter to True in as_msg_handler.py
- Replace angle brackets with square brackets for block formatting in as_msg_stat.py
- Add newline replacement in text truncation method in as_msg_stat.py
- Add loading duration timing to embedding cache loading in base_embedding_model.py
- Replace XML-style tags with markdown headers in compactor.py conversation format
- Update compactor.yaml prompts to reference markdown-style headers instead of XML tags
- Modify summarizer.py to use markdown-style conversation header format
* refactor(file-watcher): replace scan_on_start with rebuild_index_on_start parameter
- Replace scan_on_start and clear_on_start boolean parameters with single rebuild_index_on_start
- Update BaseFileWatcher constructor to use rebuild_index_on_start instead of two separate flags
- Modify initialization logic to clear and rescan when rebuild_index_on_start is True
- Remove scan_on_start parameter from CLI and light configuration files
- Update documentation to remove scan_on_start from quick start guides
- Rename all test methods and classes from scan_on_start to rebuild_index_on_start
- Add timezone-aware datetime helper method to summarizer component
- Format log message with proper line breaks for readability
* fix(core): resolve file watcher initialization issue and update version
- Fixed file watcher task creation to properly handle rebuild index on start logic
- Moved initialization and watch loop into async function to ensure proper execution order
- Updated package version from 0.3.1.1 to 0.3.1.2
- Added missing comma in embedding model logging statement
* fix(core): reduce max formatter text length limit
- Changed _DEFAULT_MAX_FORMATTER_TEXT_LENGTH from 2000 to 1000
- Updated constant value in as_msg_stat.py schema module
* fix(file-watcher): change default rebuild index behavior on start
- Changed rebuild_index_on_start parameter default from False to True
- This ensures index is rebuilt by default when file watcher starts
- Maintains consistent state initialization for file watching operations
* feat(compactor): add return_dict option and improve summary validation
- Add _is_valid_summary function to validate summary content format
- Introduce return_dict parameter to return structured results with validation
- Update prompt templates with clearer task descriptions and formatting rules
- Refactor update_user_message prompts to combine prefix and suffix logic
- Return dictionary with user_message, history_compact, and is_valid fields when enabled
- Add proper error handling for exception cases in memory compaction
- Maintain backward compatibility with string return when return_dict=False
* feat(memory): add thinking block configuration option
- Add add_thinking_block parameter to compactor component
- Pass include_thinking flag to message formatting in compactor
- Add add_thinking_block parameter to reme_light compact function
- Add add_thinking_block parameter to reme_light summarize function
- Add add_thinking_block parameter to summarizer component
- Pass include_thinking flag to message formatting in summarizer
- Remove previous-summary tags from compressed summary format
* update
* refactor(memory): remove unnecessary type check and update error logging
* refactor(core): standardize logger import and update agentscope dependency
* fix(memory): disable console output and add logging for summarizer component
* feat(core): replace OpenAI token counter with custom ReMe token counter
- Replace OpenAITokenCounter with ReMeTokenCounter implementation
- Add support for HuggingFace mirror and configurable tokenizer
- Register ReMeTokenCounter as default token counter in registry
- Update config to use hf backend with Qwen2.5-7B-Instruct model
refactor(memory): convert token counting methods to async in message handlers
- Change count_str_token, stat_message, count_msgs_token to async methods
- Update format_msgs_to_str and context_check to use async token counting
- Modify _format_tool_result_output to support async token counting
- Adjust all dependent methods to await async token counting calls
feat(memory): add dialog persistence to in-memory storage
- Implement _append_messages_to_dialog for saving messages to JSONL files
- Add dialog_path parameter to ReMeInMemoryMemory constructor
- Persist messages to daily JSONL files based on timestamp grouping
- Update mark_messages_compressed to save and remove compressed messages
- Modify clear_content to persist all messages before clearing memory
refactor(ops): update token counter type hints and initialization
- Change BaseOp to use HuggingFaceTokenCounter instead of TokenCounterBase
- Update type annotations for as_token_counter property and parameters
- Remove direct token counter injection from Compactor and ContextChecker
- Pass as_token_counter parameter through service context mechanism
style(logging): improve error logging with exception details
- Replace logger.error with logger.exception in browser control tool
- Change logger.error to logger.exception in memory get tool error handling
- Add proper exception logging with stack trace information
chore(config): add token counter configuration to light YAML
- Add as_token_counters section with default hf backend configuration
- Configure Qwen/Qwen2.5-7B-Instruct model with mirror support enabled
- Set up pretrained_model_name_or_path and use_mirror parameters
test(context): update context check tests to async implementation
- Convert verify_context_check_invariants to async function
- Update context check test methods to use async calls
- Change stat_message calls to await async implementation
- Modify test_empty_messages and test_below_threshold_returns_all to async
* feat(core): implement context checking and memory management features
* refactor(core): replace direct loguru import with logger utility function
* refactor(reme): remove RuntimeContext dependency and simplify context checking
* feat(docs): add raw conversation persistence to ReMe framework
* feat(memory): add ContextChecker component for context size management
* refactor(memory): restructure file-based memory tools and update imports
* docs(readme): update documentation with detailed architecture and components
* docs(readme): update Chinese documentation with enhanced memory management diagrams
* refactor(cookbook): move cookbook files to test directory and clean up docs
* docs(readme): update link path for old version documentation
* docs(readme): update documentation with improved architecture diagrams and component details
* docs(readme): update documentation with improved clarity and structure
* refactor(docs): update in-memory memory documentation
* docs(readme): add experiment reproduction link to quickstart guide
