feat: add Claude Code marketplace plugin with service and expert tiers
Add comprehensive Claude Code marketplace plugin supporting two paradigms
for managing markdown vaults:
- reme-service: Service-tier plugin with high-level MCP tools
(retrieve/remember/maintain) where reme2 internals handle R-M-W loop
- reme-expert: Expert-tier plugin where Claude Code agent runs R-M-W loop
directly using raw memory_* primitives guided by reme protocol skill
Both plugins implement identical 4-phase work paradigm:
- Recall: Retrieve relevant context with ranking by relevance/proximity
- Log: Record event facts and raw materials via idempotent event-folder upsert
- Distill: Promote events to topic graph via R-M-W loop
- Maintain: Vault hygiene sweep with lint and decay operations
Include marketplace configuration, documentation, MCP server setup,
subagents (reme-distiller, reme-curator), hooks (PreCompact, SessionEnd,
Stop), and slash commands (/reme-distill, /reme-recall, /reme-clean).
Remove outdated plugin entries from gitignore.
```
feat(file_watcher): add directory deletion support with descendant indexing
Add support for deleting entire directories and their indexed descendants
in the file watcher. Previously only individual file deletions were
handled properly. Now when a directory is deleted, the system finds all
indexed files beneath that directory path and removes them along with
their metadata and chunks.
The implementation includes:
- New `_descendant_indexed_paths` method to find all indexed files
under a given directory path
- Updated `_on_deleted` method to process both the target path and
all its indexed descendants
- Proper handling of symlinks and path resolution differences
- Enhanced logging to show directory deletion with child count
Also adds necessary os import for path operations.
refactor(config): restructure configuration profiles for clarity
Rename curated.yaml to remove outdated configuration file and
rename full.yaml to expert.yaml with updated documentation.
Add new service.yaml configuration profile that provides a
service-aligned MCP surface with three main tools:
- retrieve: graph-aware hybrid retrieval
- remember: single write entry point with log/distill modes
- maintain: vault hygiene sweep
The expert configuration now excludes the ingest tool since
cold-path operations are handled by external agents, and adds
memory_lint tool for structural issue detection.
Updated documentation to clarify the different configuration
profiles and their intended usage patterns.
```
docs: add ReMe2 architecture design documentation
- Add comprehensive design document (reme2.md) detailing the
three-layer architecture (L1/L2/L3) for the vault system
- Document new protocols for folder notes and memory management
- Specify interface contracts for memory_* and vault_* tools
- Outline implementation phases from current state to target
refactor: fix typo in personal retriever class
- Correct spelling error: 'retri eved_nodes' -> 'retrieved_nodes'
in PersonalRetriever.result assignment
chore: update gitignore with vault-related patterns
- Add '/vault' to ignore vault directory
- Add '/reme-plugin' to ignore plugin files
- Add '/reme2/vault' to ignore new vault implementation
```
refactor(component): rename file_store to chunk_store and update interfaces
- Rename BaseFileStore to BaseChunkStore and update component type
- Replace file_store property with chunk_store in BaseStep
- Add file_graph property to access file metadata from FileWatcher
- Update all storage backends (Chroma, Local, SQLite) to use chunk-focused APIs
- Remove file metadata handling from chunk stores (moved to FileGraph)
- Update search methods to use ChunkFilter instead of SearchFilter
- Remove file_store imports and add chunk_store imports
```
- Create MCPService class to expose jobs as MCP tools
- Implement FastMCP integration with async lifespan management
- Add environment variable setup for service information
- Implement tool registration for job execution via FunctionTool
- Support multiple transport types (sse, stdio) with configurable host/port
- Integrate with application lifecycle for proper startup/shutdown
- replace watch_paths parameter with single watch_path string
- add FileGraph integration for tracking file relationships
- implement persistent graph storage in .reme directory
- add load/build graph functionality on start
- refactor file change handling with separate methods for add/modify/delete
- update scan_existing_files to work with single path
- add file filter to exclude meta directory from watching
- remove abstract method requirement and implement concrete change handlers
BREAKING CHANGE: watch_paths parameter changed to watch_path (single path)
- Introduce BaseFileParser abstract class with component registration
- Add MdFileParser implementation for markdown files with YAML frontmatter
- Create TextFileParser implementation with built-in chunking support
- Implement file suffix enumeration for parser type safety
- Add chunking logic with configurable token size and overlap
- Support text file parsing with error handling for encoding issues
- Include line number tracking and content hashing for file chunks
- Removed hash, size, content, and chunk_count fields from FileMetadata model
- Updated chroma and local file stores to remove excluded fields from model dump
- Removed hash calculation and content storage from default file parser
- Removed chunk count tracking from file parsing logic
- Simplified FileMetadata schema to only include essential fields
- Added new SQLite file store implementation with vector and full-text search
- Added memory search step for semantic search functionality
- Remove abstract methods from base component start/close
- Update BaseJob to remove name parameter and simplify initialization
- Change file modification time field from mtime_ms to modified_time in seconds
- Add type checking imports and improve typing annotations
- Implement LocalFileStore with JSONL persistence for file chunks
- Add MdFileParser with markdown and frontmatter support
- Simplify HttpClient call method with proper kwargs handling
- Remove unused ReMe class methods and create backup version
- Update StreamJob to use step_components instead of steps attribute
- Introduce Application class for managing application lifecycle
- Add base component classes for LLM formatters and token counters
- Implement embedding model base with caching and batching support
- Create file watcher base with watchfiles integration
- Add job and step base components for workflow execution
- Update base component with async locks and improved lifecycle management
- Register new component types in component registry
- Add application context and runtime context for dependency injection
- Integrate AnthropicChatModel with new AnthropicAsLLM component
- Add component formatters for OpenAI and Anthropic chat models
- Implement token counter component with estimated token counting
- Create base client component for ReMe service communication
- Refactor BaseComponent to remove app_context parameter from _start/_close
- Update embedding model base class to remove retry logic and use npz cache
- Add job component for sequential step execution with BaseJob
- Implement step component base class for LLM workflow execution
- Enhance application context with proper type annotations
- Update component initialization to pass app_context automatically
- Remove asyncio dependency from embedding model cache operations
feat(file-store): add Chroma and SQLite file store implementations
- Add ChromaFileStore and SqliteFileStore classes to support additional
storage backends
- Export new store classes in __init__.py module
refactor(file-store): enhance BaseFileStore with hybrid search capabilities
- Add keyword scoring utility with word-match ratio and phrase bonus
- Implement hybrid search method that combines vector and keyword results
- Add merge logic for combining vector and keyword search results with
weighted scoring
- Move abstract methods to separate section for better organization
- Remove redundant search filter parameter from local implementation
refactor(file-store): simplify LocalFileStore implementation
- Remove unused delete_file_chunks and upsert_chunks methods
- Remove redundant update_file_metadata method
- Update chunk counting logic to use file_meta directly
- Simplify keyword search to use new base class scoring utility
- Remove duplicate hybrid search implementation since it's now in base class
```
- Add file_parser component with default implementation
- Introduce SearchFilter schema for path and tag filtering
- Implement filter functionality in BaseFileStore and LocalFileStore
- Update file watcher to use parser-based filtering instead of suffix filters
- Register new FILE_PARSER component enum
- Add test_data directory to gitignore
refactor: improve component imports and initialization
- Fix relative imports in application.py
- Add file_parser import to component init
- Initialize registry dict when component type doesn't exist
- Remove circular import in HttpService by using string annotation
- Update config yaml to use proper component names
refactor: enhance file watcher architecture
- Replace MdFileWatcher with more flexible FullFileWatcher and LightFileWatcher
- Remove suffix-based filtering in favor of parser-based approach
- Update BaseFileWatcher to resolve parsers from app context
- Remove unused watch_filter method
refactor: update ReMe core functionality
- Remove memory_path creation
- Simplify dream and proactive methods to return empty strings
- Update config defaults for HTTP service and component backends
docs: update component configuration in paw.yaml
- Change service backend from cmd to http
- Rename components to use correct singular forms
- Add default file parser and file watcher configurations
- Set up local file store with default settings
```
Co-authored-by: huangsen <huangsen.huang@alibaba-inc.com>
- Changed condition from divisor equals zero to divisor less than or equal to zero
- Prevents division by zero error when estimate_divisor is negative
- Maintains validation logic for invalid divisor values
- Introduce BaseAsTokenCounter and EstimatedAsTokenCounter for token estimation
- Add AsMsgStat and AsBlockStat schema for message statistics tracking
- Implement FileIO class with read/write/append/edit operations
- Create file utility functions for safe async file reading and truncation
- Add MemorySearch component for semantic search in memory files
- Register new component types in ComponentEnum and update imports
- Add constants for default host, port, and truncation limits
- Create BaseService abstract base class for service implementations
- Implement BaseStep with component accessors and lifecycle management
- Add proper __all__ exports for all new modules and components
- Introduce BaseAsTokenCounter and EstimatedAsTokenCounter for token estimation
- Add AsMsgStat and AsBlockStat schema for message statistics tracking
- Implement FileIO class with read/write/append/edit operations
- Create file utility functions for safe async file reading and truncation
- Add MemorySearch component for semantic search in memory files
- Register new component types in ComponentEnum and update imports
- Add constants for default host, port, and truncation limits
- Create BaseService abstract base class for service implementations
- Implement BaseStep with component accessors and lifecycle management
- Add proper __all__ exports for all new modules and components
- Removed unused-argument pylint disable configuration
- Cleaned up tool.pylint.messages_control section from pyproject.toml
- Updated test configuration to use explicit ignore patterns instead of pylint disables
- Add BaseClient, BaseFileStore, BaseFileWatcher, BaseJob, BaseService, and BaseStep classes
- Move component initialization logic from ApplicationContext to Application class
- Add logo printing and logging initialization in Application startup
- Create client module with base client implementation
- Add file store base class with embedding resolution and validation
- Implement file watcher base class with watchfiles integration
- Add job base class for sequential step execution orchestration
- Create service base class for job exposure mechanisms
- Refactor BaseStep with LLM workflow execution capabilities
- Add case converter utility for naming convention transformations
- Update import structure and module organization
- Add proper type hints and docstrings across all components
- Implement component registry integration for dynamic loading
- Add error handling for missing backend configurations
- Replace ReMeClient with modular client implementations
- Add component registry with type-based registration system
- Introduce BaseClient extending BaseComponent with lifecycle management
- Create HttpClient with environment-based service discovery
- Add constants for default host/port configurations
- Update service info propagation through environment variables
- Restructure imports and exports across component modules
- Add run_coro_safely utility for safe coroutine execution
- Implement component type enumeration for better organization
- Register components with R decorator for automatic discovery
- Add placeholder methods for ReMe core functionalities
- Update command-line entry point to use dynamic client selection
- Implement BaseComponent with async lifecycle and context management
- Add ApplicationContext for managing component initialization and registry
- Create Application class for orchestrating job execution and lifecycle
- Add AS LLM components with OpenAI chat model wrapper
- Implement AS LLM formatter components with OpenAI formatter
- Add client implementations including base, HTTP and ReMe clients
- Create embedding model base class with caching and batching support
- Implement file store base class with vector and full-text search
- Add file watcher components for monitoring file system changes
- Create job components for executing workflows
- Implement service components for exposing jobs via different protocols
- Add configuration schema with ApplicationConfig and ComponentConfig
- Include utility modules for case conversion, chunking, logging and similarity
- Register component types and create component registry system
* fix(core): handle chromadb import error gracefully
- Changed CHROMADB_AVAILABLE flag to _CHROMADB_IMPORT_ERROR exception storage
- Updated version from 0.3.1.7 to 0.3.1.8
- Modified import error handling to preserve original exception details
- Removed hardcoded ImportError message in favor of dynamic exception raising
- Added proper logger initialization using get_logger utility
* refactor(file_store): move sqlite3 imports inside initialization methods
- Moved sqlite3 import from module level to inside init methods
- Removed unused import statement at top of file
- Maintains same functionality while improving import organization
- Prevents potential issues with early sqlite3 dependency loading
* refactor(core): update import error handling with broader exception types
- Changed ImportError to Exception for ray import error handling
- Updated chromadb import error to use Exception instead of ImportError
- Modified elasticsearch import error to catch general exceptions
- Changed asyncpg import error handling from ImportError to Exception
- Updated qdrant import error to use Exception instead of ImportError
- Added explicit type hints for all import error variables as Exception | None
- Changed CHROMADB_AVAILABLE flag to _CHROMADB_IMPORT_ERROR exception storage
- Updated version from 0.3.1.7 to 0.3.1.8
- Modified import error handling to preserve original exception details
- Removed hardcoded ImportError message in favor of dynamic exception raising
- Added proper logger initialization using get_logger utility
* feat(compactor): add extra instruction support and improve error handling
- Add extra_instruction parameter to compactor for custom guidance during message compaction
- Implement try-catch blocks around AS LLM initialization with detailed error logging
- Add extra_instruction parameter to ReMe.compact method with comprehensive documentation
- Update agentscope dependency from 1.0.17 to 1.0.18 in light installation
- Bump version number from 0.3.1.6 to 0.3.1.7
- Pass extra_instruction parameter through compactor instantiation and execution flow
* fix(core): add error handling for AS LLM formatters and token counters initialization
- Wrapped AS LLM formatters initialization in try-except blocks
- Added specific error logging for failed AS LLM formatter initialization
- Wrapped AS token counters initialization in try-except blocks
- Added specific error logging for failed AS token counter initialization
- Applied same error handling pattern to both initial setup and restart operations
- Maintained existing warning logs for unsupported backends
* docs(context): add comprehensive context management design documentation
- Create detailed Chinese documentation for CoPaw context management V2
- Document memory layer and file system cache architecture
- Explain Pre-Reasoning Hook workflow with four-step process
- Detail two-stage truncation strategy for tool results
- Add examples for Browser Use and ReadFile tools
- Include Mermaid diagrams for visual flow representation
- Update README with link to new context design document
- Fix minor formatting issues in existing documentation
- Add protection thresholds for Markdown files in truncation
- Document long-term memory trigger mechanisms
* docs(README): add latest articles section and CoPaw context management design doc
- Added "Latest Articles" section to README with table format
- Included link to CoPaw Context Management Design document
- Created comprehensive documentation for CoPaw context management V2
- Documented in-memory and file system layer architecture
- Explained pre-reasoning hook and context compaction process
- Detailed two-phase truncation strategy for tool results
- Described special handling for readFile tool and markdown files
- Added long-term memory trigger logic overview
- Included mermaid diagrams for visualizing context flow
- Add context data structure diagram showing compact_summary and file system cache
- Update ToolResultCompactor section with detailed truncation strategies for recent vs old messages
- Add parameter tables for tool result compaction with recent_max_bytes and old_max_bytes settings
- Update execution flow steps with detailed descriptions of each memory operation
- Add key parameters table including tool_result_compact_keep_n and memory_compact_reserve
- Include thinking enhancement feature description for summary generation quality improvement
- Update both English and Chinese README documentation consistently
* refactor(file_store): simplify ChromaDB client initialization and improve file truncation logic
- Remove shutil import and _create_chroma_client method from chroma_file_store.py
- Directly initialize ChromaDB PersistentClient in start method without retry logic
- Reduce DEFAULT_MAX_BYTES from 100KB to 50KB in file_utils.py
- Update truncation notice format to provide clearer continuation instructions
- Add _truncate_fresh and _retruncate functions for better text truncation handling
- Replace inline truncation logic with dedicated function calls in file_utils.py
- Rename skills_tool_ids to md_file_tool_ids in tool_result_compactor.py
- Update file detection logic to identify any .md files instead of only skill.md
- Create comprehensive unit tests for truncation functionality in test_truncate_text_output.py
* chore(version): bump version to 0.3.1.6
- Update __version__ from 0.3.1.5 to 0.3.1.6 in __init__.py
* fix(memory): correct line numbering and improve tool result truncation
- Changed default start_line from 0 to 1 in truncate_text_output function
- Refactored _truncate method to be a standalone method in ToolResultCompactor
- Improved tool result compaction logic to handle text blocks more efficiently
- Added detection of skill-related tool calls for special handling
- Implemented conditional byte limits based on tool type for better memory management
- Updated version number from 0.3.1.4 to 0.3.1.5
* feat(file_utils): add encoding parameter to truncate_text_output function
- Added encoding parameter with default value "utf-8" to truncate_text_output function
- Updated all encode/decode calls to use the specified encoding parameter
- Modified ToolResultCompactor to pass encoding parameter when calling truncate_text_output
- Added error handling for skill tool ID detection in message processing loop
- Fixed potential AttributeError when accessing raw_input field that might be None
* fix(file-store): handle corrupted ChromaDB initialization and improve tool result truncation
- Add shutil import for directory removal operations
- Extract ChromaDB client creation into separate _create_chroma_client method
- Implement retry mechanism with database wipe on ChromaDB initialization failure
- Add proper exception handling in tool result compaction to prevent truncation errors
- Move file writing logic outside of exception handling scope for better error management
- Add warning log when truncation fails and return original content as fallback
* refactor(core): replace text truncation utilities with new marker system
- Remove old truncate_text_utils module and its exports
- Replace TRUNCATION_MARKER_START with _TRUNCATION_NOTICE_MARKER constant
- Update as_msg_stat.py to split content using new marker format
- Modify FileIO tool to use TRUNCATION_NOTICE_MARKER for continuation hints
- Change is_truncated function checks to use marker presence detection
- Move transformers dependency from main deps to light extra dependencies
- Update tool result compactor tests to verify marker instead of is_truncated calls
* feat(file_io): enhance file operations with path resolution and append functionality
- Add expanduser() to resolve file paths with ~ symbol
- Implement proper file existence and type validation in update_file
- Add new append_file method to append content to files
- Update truncation notice format for better readability
- Fix typo in error message from "provide" to "provided"
- Update transformers dependency in pyproject.toml
- Remove duplicate transformers dependency from light extras
* refactor(file_io): disable pylint too-many-return-statements warning
* perf(file_watcher): increase default polling delay and optimize watcher configuration
- Increased default poll_delay_ms from 1000ms to 2000ms to reduce CPU usage
- Removed force_polling parameter as it's no longer needed with updated polling strategy
- Simplified async watch configuration by removing conditional force_polling logic
- Reduced overall system resource consumption during file watching operations
* refactor(memory): update conversation log documentation in memory summary
- Changed "Raw conversation logs" to "Earlier conversation logs" for clarity
- Added warning note about potentially large dialog file sizes
- Improved formatting with additional line break for better readability
- Maintained existing compressed summary integration unchanged
* feat(memory): add long-term memory support to file-based memory system
- Initialize _long_term_memory attribute as empty string
- Add memories section to content when long-term memory exists
- Consolidate summary and memories into single user message
- Format memories with markdown header # Memories
- Maintain existing compressed summary functionality
- Join multiple content parts with double newlines
* chore(deps): update version and move litellm to dev dependencies
- Updated package version from 0.3.1.2 to 0.3.1.3
- Removed litellm from main dependencies in pyproject.toml
- Added litellm as fixed version dependency in dev group
- Maintained litellm requirement while reorganizing dependency structure
* chore(deps): move litellm dependency to full extras
- Moved litellm==1.80.0 from main dependencies to full extra
- Kept litellm as optional dependency for users needing full feature set
- Maintains backward compatibility for light installation option
* feat(pyproject): add litellm dependency to project configuration
- Added litellm==1.80.0 as optional dependency in pyproject.toml
- Created new litellm extra group for LiteLLM integration
- Updated full dependency group to include the new litellm option
* style(memory): update message formatting and improve logging
- Change default include_thinking parameter to True in as_msg_handler.py
- Replace angle brackets with square brackets for block formatting in as_msg_stat.py
- Add newline replacement in text truncation method in as_msg_stat.py
- Add loading duration timing to embedding cache loading in base_embedding_model.py
- Replace XML-style tags with markdown headers in compactor.py conversation format
- Update compactor.yaml prompts to reference markdown-style headers instead of XML tags
- Modify summarizer.py to use markdown-style conversation header format
* refactor(file-watcher): replace scan_on_start with rebuild_index_on_start parameter
- Replace scan_on_start and clear_on_start boolean parameters with single rebuild_index_on_start
- Update BaseFileWatcher constructor to use rebuild_index_on_start instead of two separate flags
- Modify initialization logic to clear and rescan when rebuild_index_on_start is True
- Remove scan_on_start parameter from CLI and light configuration files
- Update documentation to remove scan_on_start from quick start guides
- Rename all test methods and classes from scan_on_start to rebuild_index_on_start
- Add timezone-aware datetime helper method to summarizer component
- Format log message with proper line breaks for readability
* fix(core): resolve file watcher initialization issue and update version
- Fixed file watcher task creation to properly handle rebuild index on start logic
- Moved initialization and watch loop into async function to ensure proper execution order
- Updated package version from 0.3.1.1 to 0.3.1.2
- Added missing comma in embedding model logging statement
* fix(core): reduce max formatter text length limit
- Changed _DEFAULT_MAX_FORMATTER_TEXT_LENGTH from 2000 to 1000
- Updated constant value in as_msg_stat.py schema module
* fix(file-watcher): change default rebuild index behavior on start
- Changed rebuild_index_on_start parameter default from False to True
- This ensures index is rebuilt by default when file watcher starts
- Maintains consistent state initialization for file watching operations
* feat(compactor): add return_dict option and improve summary validation
- Add _is_valid_summary function to validate summary content format
- Introduce return_dict parameter to return structured results with validation
- Update prompt templates with clearer task descriptions and formatting rules
- Refactor update_user_message prompts to combine prefix and suffix logic
- Return dictionary with user_message, history_compact, and is_valid fields when enabled
- Add proper error handling for exception cases in memory compaction
- Maintain backward compatibility with string return when return_dict=False
* feat(memory): add thinking block configuration option
- Add add_thinking_block parameter to compactor component
- Pass include_thinking flag to message formatting in compactor
- Add add_thinking_block parameter to reme_light compact function
- Add add_thinking_block parameter to reme_light summarize function
- Add add_thinking_block parameter to summarizer component
- Pass include_thinking flag to message formatting in summarizer
- Remove previous-summary tags from compressed summary format
