- Replace legacy line-based chunking with AST tree approach that builds
a complete document skeleton with content inlined under relevant
sections
- Add new chunking parameters: chunk_chars (default 2000) and embed_toc
(default True) to control content size and TOC inclusion
- Implement recursive chunking algorithm that respects structural
boundaries (code lines, table rows, list items) and prevents splits
inside blocks
- Introduce part markers [Part X/N] for oversized leaf blocks that
require splitting
- Add CLI tool for inspecting parsed chunks and edges with options for
preview and configuration
- Refactor edge extraction to use FileEdge.from_text instead of
parse_wikilinks for consistency
BREAKING CHANGE: Chunk format changes significantly with full TOC
skeleton wrapping content, affecting embedding models expecting
breadcrumb prefixes.
Remove SqliteFileStore backend and simplify the base file store
interface. The file store now focuses on in-memory + JSONL persistence
for the (file → chunks) graph with reduced complexity.
BREAKING CHANGE: SqliteFileStore is removed, only LocalFileStore remains
available.
docs(protocol): add typed edge link protocol documentation
Add comprehensive documentation for the link protocol supporting
typed edges in body text. This includes specification for three
legal inline forms (bare wikilink, line-level Dataview,
inline-bracketed Dataview), predicate syntax rules, and the
machine-managed Relations section convention for organizing
discovered edges.
fix(memory): update path reference from vault_root to working_dir
Change the memory_create operation's path anchoring from
vault_root to working_dir to maintain consistency with the
current working directory configuration.
refactor(components): remove edge_extractor module and simplify parsing
Remove the edge_extractor component module entirely and
inline edge extraction logic directly into LinkedFileParser
using parse_wikilinks utility. This simplifies the architecture
by eliminating the separate edge extraction component and
delegating edge discovery to the maintainer's enrichment operations.
feat(parser): update parse method signature and simplify edge extraction
Modify LinkedFileParser to return (FileNode, list[FileChunk])
tuple instead of ParsedFile, remove dependency on BaseEdgeExtractor,
and implement direct wikilink parsing from body text only.
```
feat: add Claude Code marketplace plugin with service and expert tiers
Add comprehensive Claude Code marketplace plugin supporting two paradigms
for managing markdown vaults:
- reme-service: Service-tier plugin with high-level MCP tools
(retrieve/remember/maintain) where reme2 internals handle R-M-W loop
- reme-expert: Expert-tier plugin where Claude Code agent runs R-M-W loop
directly using raw memory_* primitives guided by reme protocol skill
Both plugins implement identical 4-phase work paradigm:
- Recall: Retrieve relevant context with ranking by relevance/proximity
- Log: Record event facts and raw materials via idempotent event-folder upsert
- Distill: Promote events to topic graph via R-M-W loop
- Maintain: Vault hygiene sweep with lint and decay operations
Include marketplace configuration, documentation, MCP server setup,
subagents (reme-distiller, reme-curator), hooks (PreCompact, SessionEnd,
Stop), and slash commands (/reme-distill, /reme-recall, /reme-clean).
Remove outdated plugin entries from gitignore.
```
feat(file_watcher): add directory deletion support with descendant indexing
Add support for deleting entire directories and their indexed descendants
in the file watcher. Previously only individual file deletions were
handled properly. Now when a directory is deleted, the system finds all
indexed files beneath that directory path and removes them along with
their metadata and chunks.
The implementation includes:
- New `_descendant_indexed_paths` method to find all indexed files
under a given directory path
- Updated `_on_deleted` method to process both the target path and
all its indexed descendants
- Proper handling of symlinks and path resolution differences
- Enhanced logging to show directory deletion with child count
Also adds necessary os import for path operations.
refactor(config): restructure configuration profiles for clarity
Rename curated.yaml to remove outdated configuration file and
rename full.yaml to expert.yaml with updated documentation.
Add new service.yaml configuration profile that provides a
service-aligned MCP surface with three main tools:
- retrieve: graph-aware hybrid retrieval
- remember: single write entry point with log/distill modes
- maintain: vault hygiene sweep
The expert configuration now excludes the ingest tool since
cold-path operations are handled by external agents, and adds
memory_lint tool for structural issue detection.
Updated documentation to clarify the different configuration
profiles and their intended usage patterns.
```
docs: add ReMe2 architecture design documentation
- Add comprehensive design document (reme2.md) detailing the
three-layer architecture (L1/L2/L3) for the vault system
- Document new protocols for folder notes and memory management
- Specify interface contracts for memory_* and vault_* tools
- Outline implementation phases from current state to target
refactor: fix typo in personal retriever class
- Correct spelling error: 'retri eved_nodes' -> 'retrieved_nodes'
in PersonalRetriever.result assignment
chore: update gitignore with vault-related patterns
- Add '/vault' to ignore vault directory
- Add '/reme-plugin' to ignore plugin files
- Add '/reme2/vault' to ignore new vault implementation
```
refactor(component): rename file_store to chunk_store and update interfaces
- Rename BaseFileStore to BaseChunkStore and update component type
- Replace file_store property with chunk_store in BaseStep
- Add file_graph property to access file metadata from FileWatcher
- Update all storage backends (Chroma, Local, SQLite) to use chunk-focused APIs
- Remove file metadata handling from chunk stores (moved to FileGraph)
- Update search methods to use ChunkFilter instead of SearchFilter
- Remove file_store imports and add chunk_store imports
```
- Create MCPService class to expose jobs as MCP tools
- Implement FastMCP integration with async lifespan management
- Add environment variable setup for service information
- Implement tool registration for job execution via FunctionTool
- Support multiple transport types (sse, stdio) with configurable host/port
- Integrate with application lifecycle for proper startup/shutdown
- replace watch_paths parameter with single watch_path string
- add FileGraph integration for tracking file relationships
- implement persistent graph storage in .reme directory
- add load/build graph functionality on start
- refactor file change handling with separate methods for add/modify/delete
- update scan_existing_files to work with single path
- add file filter to exclude meta directory from watching
- remove abstract method requirement and implement concrete change handlers
BREAKING CHANGE: watch_paths parameter changed to watch_path (single path)
- Introduce BaseFileParser abstract class with component registration
- Add MdFileParser implementation for markdown files with YAML frontmatter
- Create TextFileParser implementation with built-in chunking support
- Implement file suffix enumeration for parser type safety
- Add chunking logic with configurable token size and overlap
- Support text file parsing with error handling for encoding issues
- Include line number tracking and content hashing for file chunks
- Removed hash, size, content, and chunk_count fields from FileMetadata model
- Updated chroma and local file stores to remove excluded fields from model dump
- Removed hash calculation and content storage from default file parser
- Removed chunk count tracking from file parsing logic
- Simplified FileMetadata schema to only include essential fields
- Added new SQLite file store implementation with vector and full-text search
- Added memory search step for semantic search functionality
- Remove abstract methods from base component start/close
- Update BaseJob to remove name parameter and simplify initialization
- Change file modification time field from mtime_ms to modified_time in seconds
- Add type checking imports and improve typing annotations
- Implement LocalFileStore with JSONL persistence for file chunks
- Add MdFileParser with markdown and frontmatter support
- Simplify HttpClient call method with proper kwargs handling
- Remove unused ReMe class methods and create backup version
- Update StreamJob to use step_components instead of steps attribute
- Introduce Application class for managing application lifecycle
- Add base component classes for LLM formatters and token counters
- Implement embedding model base with caching and batching support
- Create file watcher base with watchfiles integration
- Add job and step base components for workflow execution
- Update base component with async locks and improved lifecycle management
- Register new component types in component registry
- Add application context and runtime context for dependency injection
- Integrate AnthropicChatModel with new AnthropicAsLLM component
- Add component formatters for OpenAI and Anthropic chat models
- Implement token counter component with estimated token counting
- Create base client component for ReMe service communication
- Refactor BaseComponent to remove app_context parameter from _start/_close
- Update embedding model base class to remove retry logic and use npz cache
- Add job component for sequential step execution with BaseJob
- Implement step component base class for LLM workflow execution
- Enhance application context with proper type annotations
- Update component initialization to pass app_context automatically
- Remove asyncio dependency from embedding model cache operations
feat(file-store): add Chroma and SQLite file store implementations
- Add ChromaFileStore and SqliteFileStore classes to support additional
storage backends
- Export new store classes in __init__.py module
refactor(file-store): enhance BaseFileStore with hybrid search capabilities
- Add keyword scoring utility with word-match ratio and phrase bonus
- Implement hybrid search method that combines vector and keyword results
- Add merge logic for combining vector and keyword search results with
weighted scoring
- Move abstract methods to separate section for better organization
- Remove redundant search filter parameter from local implementation
refactor(file-store): simplify LocalFileStore implementation
- Remove unused delete_file_chunks and upsert_chunks methods
- Remove redundant update_file_metadata method
- Update chunk counting logic to use file_meta directly
- Simplify keyword search to use new base class scoring utility
- Remove duplicate hybrid search implementation since it's now in base class
```
- Add file_parser component with default implementation
- Introduce SearchFilter schema for path and tag filtering
- Implement filter functionality in BaseFileStore and LocalFileStore
- Update file watcher to use parser-based filtering instead of suffix filters
- Register new FILE_PARSER component enum
- Add test_data directory to gitignore
refactor: improve component imports and initialization
- Fix relative imports in application.py
- Add file_parser import to component init
- Initialize registry dict when component type doesn't exist
- Remove circular import in HttpService by using string annotation
- Update config yaml to use proper component names
refactor: enhance file watcher architecture
- Replace MdFileWatcher with more flexible FullFileWatcher and LightFileWatcher
- Remove suffix-based filtering in favor of parser-based approach
- Update BaseFileWatcher to resolve parsers from app context
- Remove unused watch_filter method
refactor: update ReMe core functionality
- Remove memory_path creation
- Simplify dream and proactive methods to return empty strings
- Update config defaults for HTTP service and component backends
docs: update component configuration in paw.yaml
- Change service backend from cmd to http
- Rename components to use correct singular forms
- Add default file parser and file watcher configurations
- Set up local file store with default settings
```
Co-authored-by: huangsen <huangsen.huang@alibaba-inc.com>
- Changed condition from divisor equals zero to divisor less than or equal to zero
- Prevents division by zero error when estimate_divisor is negative
- Maintains validation logic for invalid divisor values
- Introduce BaseAsTokenCounter and EstimatedAsTokenCounter for token estimation
- Add AsMsgStat and AsBlockStat schema for message statistics tracking
- Implement FileIO class with read/write/append/edit operations
- Create file utility functions for safe async file reading and truncation
- Add MemorySearch component for semantic search in memory files
- Register new component types in ComponentEnum and update imports
- Add constants for default host, port, and truncation limits
- Create BaseService abstract base class for service implementations
- Implement BaseStep with component accessors and lifecycle management
- Add proper __all__ exports for all new modules and components
- Introduce BaseAsTokenCounter and EstimatedAsTokenCounter for token estimation
- Add AsMsgStat and AsBlockStat schema for message statistics tracking
- Implement FileIO class with read/write/append/edit operations
- Create file utility functions for safe async file reading and truncation
- Add MemorySearch component for semantic search in memory files
- Register new component types in ComponentEnum and update imports
- Add constants for default host, port, and truncation limits
- Create BaseService abstract base class for service implementations
- Implement BaseStep with component accessors and lifecycle management
- Add proper __all__ exports for all new modules and components
- Removed unused-argument pylint disable configuration
- Cleaned up tool.pylint.messages_control section from pyproject.toml
- Updated test configuration to use explicit ignore patterns instead of pylint disables
- Add BaseClient, BaseFileStore, BaseFileWatcher, BaseJob, BaseService, and BaseStep classes
- Move component initialization logic from ApplicationContext to Application class
- Add logo printing and logging initialization in Application startup
- Create client module with base client implementation
- Add file store base class with embedding resolution and validation
- Implement file watcher base class with watchfiles integration
- Add job base class for sequential step execution orchestration
- Create service base class for job exposure mechanisms
- Refactor BaseStep with LLM workflow execution capabilities
- Add case converter utility for naming convention transformations
- Update import structure and module organization
- Add proper type hints and docstrings across all components
- Implement component registry integration for dynamic loading
- Add error handling for missing backend configurations
- Replace ReMeClient with modular client implementations
- Add component registry with type-based registration system
- Introduce BaseClient extending BaseComponent with lifecycle management
- Create HttpClient with environment-based service discovery
- Add constants for default host/port configurations
- Update service info propagation through environment variables
- Restructure imports and exports across component modules
- Add run_coro_safely utility for safe coroutine execution
- Implement component type enumeration for better organization
- Register components with R decorator for automatic discovery
- Add placeholder methods for ReMe core functionalities
- Update command-line entry point to use dynamic client selection
- Implement BaseComponent with async lifecycle and context management
- Add ApplicationContext for managing component initialization and registry
- Create Application class for orchestrating job execution and lifecycle
- Add AS LLM components with OpenAI chat model wrapper
- Implement AS LLM formatter components with OpenAI formatter
- Add client implementations including base, HTTP and ReMe clients
- Create embedding model base class with caching and batching support
- Implement file store base class with vector and full-text search
- Add file watcher components for monitoring file system changes
- Create job components for executing workflows
- Implement service components for exposing jobs via different protocols
- Add configuration schema with ApplicationConfig and ComponentConfig
- Include utility modules for case conversion, chunking, logging and similarity
- Register component types and create component registry system
* fix(core): handle chromadb import error gracefully
- Changed CHROMADB_AVAILABLE flag to _CHROMADB_IMPORT_ERROR exception storage
- Updated version from 0.3.1.7 to 0.3.1.8
- Modified import error handling to preserve original exception details
- Removed hardcoded ImportError message in favor of dynamic exception raising
- Added proper logger initialization using get_logger utility
* refactor(file_store): move sqlite3 imports inside initialization methods
- Moved sqlite3 import from module level to inside init methods
- Removed unused import statement at top of file
- Maintains same functionality while improving import organization
- Prevents potential issues with early sqlite3 dependency loading
* refactor(core): update import error handling with broader exception types
- Changed ImportError to Exception for ray import error handling
- Updated chromadb import error to use Exception instead of ImportError
- Modified elasticsearch import error to catch general exceptions
- Changed asyncpg import error handling from ImportError to Exception
- Updated qdrant import error to use Exception instead of ImportError
- Added explicit type hints for all import error variables as Exception | None
- Changed CHROMADB_AVAILABLE flag to _CHROMADB_IMPORT_ERROR exception storage
- Updated version from 0.3.1.7 to 0.3.1.8
- Modified import error handling to preserve original exception details
- Removed hardcoded ImportError message in favor of dynamic exception raising
- Added proper logger initialization using get_logger utility
* feat(compactor): add extra instruction support and improve error handling
- Add extra_instruction parameter to compactor for custom guidance during message compaction
- Implement try-catch blocks around AS LLM initialization with detailed error logging
- Add extra_instruction parameter to ReMe.compact method with comprehensive documentation
- Update agentscope dependency from 1.0.17 to 1.0.18 in light installation
- Bump version number from 0.3.1.6 to 0.3.1.7
- Pass extra_instruction parameter through compactor instantiation and execution flow
* fix(core): add error handling for AS LLM formatters and token counters initialization
- Wrapped AS LLM formatters initialization in try-except blocks
- Added specific error logging for failed AS LLM formatter initialization
- Wrapped AS token counters initialization in try-except blocks
- Added specific error logging for failed AS token counter initialization
- Applied same error handling pattern to both initial setup and restart operations
- Maintained existing warning logs for unsupported backends
* docs(context): add comprehensive context management design documentation
- Create detailed Chinese documentation for CoPaw context management V2
- Document memory layer and file system cache architecture
- Explain Pre-Reasoning Hook workflow with four-step process
- Detail two-stage truncation strategy for tool results
- Add examples for Browser Use and ReadFile tools
- Include Mermaid diagrams for visual flow representation
- Update README with link to new context design document
- Fix minor formatting issues in existing documentation
- Add protection thresholds for Markdown files in truncation
- Document long-term memory trigger mechanisms
* docs(README): add latest articles section and CoPaw context management design doc
- Added "Latest Articles" section to README with table format
- Included link to CoPaw Context Management Design document
- Created comprehensive documentation for CoPaw context management V2
- Documented in-memory and file system layer architecture
- Explained pre-reasoning hook and context compaction process
- Detailed two-phase truncation strategy for tool results
- Described special handling for readFile tool and markdown files
- Added long-term memory trigger logic overview
- Included mermaid diagrams for visualizing context flow
- Add context data structure diagram showing compact_summary and file system cache
- Update ToolResultCompactor section with detailed truncation strategies for recent vs old messages
- Add parameter tables for tool result compaction with recent_max_bytes and old_max_bytes settings
- Update execution flow steps with detailed descriptions of each memory operation
- Add key parameters table including tool_result_compact_keep_n and memory_compact_reserve
- Include thinking enhancement feature description for summary generation quality improvement
- Update both English and Chinese README documentation consistently
