Add mistletoe as a project dependency for enhanced markdown parsing
capabilities. Refactor the LinkedFileParser to use a proper AST-based
approach with MdNode tree structure, replacing the previous flat token
processing method. The new implementation provides better handling of
markdown elements including tables, code fences, lists, and headings,
with improved chunking logic that maintains document structure in
generated content segments.
The changes include:
- Add mistletoe dependency to pyproject.toml
- Implement proper AST node representation with MdNode class
- Create recursive chunking algorithm with TOC preservation
- Add support for frontmatter extraction with FileFrontMatter schema
- Optimize leaf node splitting with proper boundary detection
- Include part numbering for split content pieces
- Introduce BaseAsTokenCounter and EstimatedAsTokenCounter for token estimation
- Add AsMsgStat and AsBlockStat schema for message statistics tracking
- Implement FileIO class with read/write/append/edit operations
- Create file utility functions for safe async file reading and truncation
- Add MemorySearch component for semantic search in memory files
- Register new component types in ComponentEnum and update imports
- Add constants for default host, port, and truncation limits
- Create BaseService abstract base class for service implementations
- Implement BaseStep with component accessors and lifecycle management
- Add proper __all__ exports for all new modules and components
- Removed unused-argument pylint disable configuration
- Cleaned up tool.pylint.messages_control section from pyproject.toml
- Updated test configuration to use explicit ignore patterns instead of pylint disables
- Add BaseClient, BaseFileStore, BaseFileWatcher, BaseJob, BaseService, and BaseStep classes
- Move component initialization logic from ApplicationContext to Application class
- Add logo printing and logging initialization in Application startup
- Create client module with base client implementation
- Add file store base class with embedding resolution and validation
- Implement file watcher base class with watchfiles integration
- Add job base class for sequential step execution orchestration
- Create service base class for job exposure mechanisms
- Refactor BaseStep with LLM workflow execution capabilities
- Add case converter utility for naming convention transformations
- Update import structure and module organization
- Add proper type hints and docstrings across all components
- Implement component registry integration for dynamic loading
- Add error handling for missing backend configurations
- Implement BaseComponent with async lifecycle and context management
- Add ApplicationContext for managing component initialization and registry
- Create Application class for orchestrating job execution and lifecycle
- Add AS LLM components with OpenAI chat model wrapper
- Implement AS LLM formatter components with OpenAI formatter
- Add client implementations including base, HTTP and ReMe clients
- Create embedding model base class with caching and batching support
- Implement file store base class with vector and full-text search
- Add file watcher components for monitoring file system changes
- Create job components for executing workflows
- Implement service components for exposing jobs via different protocols
- Add configuration schema with ApplicationConfig and ComponentConfig
- Include utility modules for case conversion, chunking, logging and similarity
- Register component types and create component registry system
* feat(compactor): add extra instruction support and improve error handling
- Add extra_instruction parameter to compactor for custom guidance during message compaction
- Implement try-catch blocks around AS LLM initialization with detailed error logging
- Add extra_instruction parameter to ReMe.compact method with comprehensive documentation
- Update agentscope dependency from 1.0.17 to 1.0.18 in light installation
- Bump version number from 0.3.1.6 to 0.3.1.7
- Pass extra_instruction parameter through compactor instantiation and execution flow
* fix(core): add error handling for AS LLM formatters and token counters initialization
- Wrapped AS LLM formatters initialization in try-except blocks
- Added specific error logging for failed AS LLM formatter initialization
- Wrapped AS token counters initialization in try-except blocks
- Added specific error logging for failed AS token counter initialization
- Applied same error handling pattern to both initial setup and restart operations
- Maintained existing warning logs for unsupported backends
* refactor(core): replace text truncation utilities with new marker system
- Remove old truncate_text_utils module and its exports
- Replace TRUNCATION_MARKER_START with _TRUNCATION_NOTICE_MARKER constant
- Update as_msg_stat.py to split content using new marker format
- Modify FileIO tool to use TRUNCATION_NOTICE_MARKER for continuation hints
- Change is_truncated function checks to use marker presence detection
- Move transformers dependency from main deps to light extra dependencies
- Update tool result compactor tests to verify marker instead of is_truncated calls
* feat(file_io): enhance file operations with path resolution and append functionality
- Add expanduser() to resolve file paths with ~ symbol
- Implement proper file existence and type validation in update_file
- Add new append_file method to append content to files
- Update truncation notice format for better readability
- Fix typo in error message from "provide" to "provided"
- Update transformers dependency in pyproject.toml
- Remove duplicate transformers dependency from light extras
* refactor(file_io): disable pylint too-many-return-statements warning
* perf(file_watcher): increase default polling delay and optimize watcher configuration
- Increased default poll_delay_ms from 1000ms to 2000ms to reduce CPU usage
- Removed force_polling parameter as it's no longer needed with updated polling strategy
- Simplified async watch configuration by removing conditional force_polling logic
- Reduced overall system resource consumption during file watching operations
* refactor(memory): update conversation log documentation in memory summary
- Changed "Raw conversation logs" to "Earlier conversation logs" for clarity
- Added warning note about potentially large dialog file sizes
- Improved formatting with additional line break for better readability
- Maintained existing compressed summary integration unchanged
* feat(memory): add long-term memory support to file-based memory system
- Initialize _long_term_memory attribute as empty string
- Add memories section to content when long-term memory exists
- Consolidate summary and memories into single user message
- Format memories with markdown header # Memories
- Maintain existing compressed summary functionality
- Join multiple content parts with double newlines
* chore(deps): update version and move litellm to dev dependencies
- Updated package version from 0.3.1.2 to 0.3.1.3
- Removed litellm from main dependencies in pyproject.toml
- Added litellm as fixed version dependency in dev group
- Maintained litellm requirement while reorganizing dependency structure
* chore(deps): move litellm dependency to full extras
- Moved litellm==1.80.0 from main dependencies to full extra
- Kept litellm as optional dependency for users needing full feature set
- Maintains backward compatibility for light installation option
* feat(pyproject): add litellm dependency to project configuration
- Added litellm==1.80.0 as optional dependency in pyproject.toml
- Created new litellm extra group for LiteLLM integration
- Updated full dependency group to include the new litellm option
* update
* refactor(memory): remove unnecessary type check and update error logging
* refactor(core): standardize logger import and update agentscope dependency
* fix(memory): disable console output and add logging for summarizer component
* feat(core): replace OpenAI token counter with custom ReMe token counter
- Replace OpenAITokenCounter with ReMeTokenCounter implementation
- Add support for HuggingFace mirror and configurable tokenizer
- Register ReMeTokenCounter as default token counter in registry
- Update config to use hf backend with Qwen2.5-7B-Instruct model
refactor(memory): convert token counting methods to async in message handlers
- Change count_str_token, stat_message, count_msgs_token to async methods
- Update format_msgs_to_str and context_check to use async token counting
- Modify _format_tool_result_output to support async token counting
- Adjust all dependent methods to await async token counting calls
feat(memory): add dialog persistence to in-memory storage
- Implement _append_messages_to_dialog for saving messages to JSONL files
- Add dialog_path parameter to ReMeInMemoryMemory constructor
- Persist messages to daily JSONL files based on timestamp grouping
- Update mark_messages_compressed to save and remove compressed messages
- Modify clear_content to persist all messages before clearing memory
refactor(ops): update token counter type hints and initialization
- Change BaseOp to use HuggingFaceTokenCounter instead of TokenCounterBase
- Update type annotations for as_token_counter property and parameters
- Remove direct token counter injection from Compactor and ContextChecker
- Pass as_token_counter parameter through service context mechanism
style(logging): improve error logging with exception details
- Replace logger.error with logger.exception in browser control tool
- Change logger.error to logger.exception in memory get tool error handling
- Add proper exception logging with stack trace information
chore(config): add token counter configuration to light YAML
- Add as_token_counters section with default hf backend configuration
- Configure Qwen/Qwen2.5-7B-Instruct model with mirror support enabled
- Set up pretrained_model_name_or_path and use_mirror parameters
test(context): update context check tests to async implementation
- Convert verify_context_check_invariants to async function
- Update context check test methods to use async calls
- Change stat_message calls to await async implementation
- Modify test_empty_messages and test_below_threshold_returns_all to async
* feat(core): implement context checking and memory management features
* refactor(core): replace direct loguru import with logger utility function
* refactor(reme): remove RuntimeContext dependency and simplify context checking
* feat(docs): add raw conversation persistence to ReMe framework
- Add Weikang Zhou as a contributor in pyproject.toml
- Update README.md with new author in software citation
- Add new paper reference for AgentscopeReMe framework
- Include arXiv link and publication details
- Add full author list for the research paper
- Update bibliography with proper formatting
- Added QdrantVectorStore backend with native async operations
- Implemented advanced filtering capabilities for metadata queries
- Added support for Qdrant Cloud and local deployments- Updated vector store comparison table with Qdrant features
- Enhanced documentation with Qdrant setup and usage examples
- Fixed code block formatting in vector store API guide
- Updated embedding model integration for Qdrant compatibility
- Updated __version__ in reme_ai/__init__.py
- Updated project version in pyproject.toml- Changed flowllm dependency to include reme extra
- Fixed typo in README.md query example
- Enhanced tool call result parsing with improved scoring logic (0.0 or 1.0)
- Updated tool memory schema to reflect binary success/failure scoring
- Added deterministic behavior support via seed configuration in mock tools
- Improved evaluation prompts to focus on result quality over success flags
- Extended README with tool memory documentation and usage examples- Added utility functions for generating mock tool call results
- Removed deprecated test file for UseMockSearchOp- Updated default configurations to include use_mock_search operation- Bumped version to0.1.10 and updated flowllm dependency requirement
- Moved deprecation warnings to main init file
- Simplified tool memory summary formatting by removing redundant statistics
- Fixed tool call result processing to handle multiple tool names concurrently
- Change 'str' to 'string' for consistency in data types
- Rename 'list' to 'array' for JSON compatibility
- Update 'bool' to 'boolean' and 'int' to 'integer' for standardization
- Upgrade flowllm dependency from >=0.1.6 to >=0.1.7
- Update message variable from 'messages' to 'new_messages' in memory services
- Bump package version from 0.1.6 to 0.1.7
- Update FlowLLM dependency version from >=0.1.5 to >=0.1.6
