* feat(benchmark): add golden answer validation and session review for LongMemEval
- Introduce GoldenCheckStep to validate LongMemEval golden answers using structured verdicts
- Add SessionReviewStep to extract query/answer-relevant evidence from all sessions
- Implement concurrent session processing with configurable concurrency limits
- Create check_golden job configuration with lme_review and lme_judge agent wrappers
- Add Qwen3.7-plus model configuration for enhanced processing capabilities
- Include python_execute tool integration for agent-based reasoning and date validation
- Generate comprehensive JSON output with session summaries and validation verdicts
- Add run_check_golden.py script for batch processing across all LongMemEval samples
- Configure proper logging initialization with console and file output options
- Update component registry and file I/O modules to support new benchmark features
* feat(scripts): add script to summarize LongMemEval check_golden verdicts
- Parse check_golden.json files across all LongMemEval samples
- Calculate accuracy metrics for golden answers and session IDs
- Provide breakdown by question type with percentage calculations
- Add command line options for listing bad samples and JSON output
- Include progress tracking showing completed vs pending samples
- Display confidence scores and date sanity checks statistics
* refactor(benchmark): move golden check scripts to longmemeval directory
- Moved run_check_golden.py from scripts/ to benchmark/longmemeval/
- Moved stats_check_golden.py from scripts/ to benchmark/longmemeval/
- Updated path resolution to use parents[2] instead of parent.parent
- Added new --list-run-failed option to stats script
- Added logging directory constant and functions for tracking launched samples
- Enhanced stats output with launched count and run failure information
- Improved error reporting with run failure details and log file paths
* feat(benchmark): add LongMemEval agentic answer workflow with session extraction
- Add LmeAgenticAnswerStep, LmeAutoMemoryStep, and LmeExtractSessionStep to __init__.py
- Create shared helper render_with_source for displaying search results with session_id
- Implement agentic_answer step with vector_search, bm25_search, and extract_session_by_id tools
- Add auto_memory step to convert each session into search-friendly daily notes
- Create extract_session step to retrieve and analyze raw session content by session_id
- Update jinli_lme.yaml with auto_memory, vector_search, bm25_search, and agentic_answer jobs
- Configure lme_memory, lme_extract, and lme_agentic_answer agent wrappers
- Enhance search steps with include_source option to show session_id metadata
- Add proper session_id tracking and collision handling in daily note generation
* feat(benchmark): add LongMemEval agentic answer evaluation pipeline
- Add session_id tracking to agentic_answer.py result metadata
- Introduce run_agentic_answer.py driver for complete pipeline execution
- Implement auto_memory, update_index, and agentic_answer job orchestration
- Add concurrent execution with configurable limits and staggering
- Create aggregation script for collecting tool-call trails and results
- Add stats_agentic_answer.py for comprehensive result analysis
- Implement resume capability with existing output detection
- Generate aggregate.json with per-sample breakdown and tool call summaries
* feat(steps): add ClearPathsStep for cleaning workspace outputs before rebuild
- Introduce ClearPathsStep to remove stale workspace files/directories
- Add support for specifying paths and config_keys as targets to clear
- Implement safety checks to prevent deletion of files outside workspace
- Add logging for cleared paths and warnings for invalid paths
- Configure clear_paths_step in jinli_lme.yaml to clean daily_dir
- Add clear_paths_step to clean mem_answer.json before rebuilds
* feat(benchmark): add resume functionality to agentic answer runner
- Replace --force flag with --resume flag for controlling job execution
- By default every job reruns with clean rebuild behavior using config clear steps
- Add --resume option to skip samples whose output already exists and continue interrupted batches
- Update documentation to reflect new default clean rebuild behavior
- Modify job skipping logic to honor resume flag instead of force flag
- Update dry-run output to show correct todo jobs based on resume status
- Change default example command to use --resume for continuing interrupted runs
* feat(benchmark): generate JSONL output for check golden records
- Add write_check_golden_list function to create JSONL file
- Write all readable check_golden records as JSONL format
- Include check_golden_list path in stats output
- Display generated JSONL file path in summary report
- Maintain UTF-8 encoding with non-ASCII character support
* refactor(benchmark): rename answer judge step and integrate LME LLM judge
- Rename AnswerJudgeStep to LmeLlmJudgeStep and update imports
- Add new llm_judge configuration in jinli_lme.yaml
- Update run_agentic_answer.py to include llm_judge in pipeline
- Modify LmeLlmJudgeStep to read from query.json and answer.json
- Write LLM judgement results back to mem_answer.json
- Add command line options for start/end sample range selection
- Update aggregate.json generation to include LLM judgement data
- Add resume capability for llm_judge job based on judgement presence
* refactor(benchmark): rename answer judge step and integrate LME LLM judge
- Rename AnswerJudgeStep to LmeLlmJudgeStep and update imports
- Add new llm_judge configuration in jinli_lme.yaml
- Update run_agentic_answer.py to include llm_judge in pipeline
- Modify LmeLlmJudgeStep to read from query.json and answer.json
- Write LLM judgement results back to mem_answer.json
- Add command line options for start/end sample range selection
- Update aggregate.json generation to include LLM judgement data
- Add resume capability for llm_judge job based on judgement presence
* feat(steps): add wait_for_paths_step to block until workspace files exist
- Introduce WaitForPathsStep class that polls for required workspace-relative paths
- Add step registration with 'wait_for_paths_step' backend identifier
- Implement path validation to ensure targets are within workspace boundaries
- Add polling mechanism with configurable intervals via poll_seconds parameter
- Include logging functionality with log_every_seconds parameter for status updates
- Add metadata tracking of waited paths and duration in response object
- Register step in index module and expose in public API
- Configure step in jinli_lme.yaml to wait for session_review.json before golden check
- Add script rename from run_check_golden.py to run_golden_check.py with enhanced options
* feat(benchmark): enhance longmemeval benchmarking with concurrency and progress tracking
- Add benchmark extra dependency group with portalocker requirement
- Introduce concurrent execution support for golden_check and session_review workflows
- Add progress reporting interval option with real-time status updates
- Implement global throttling mechanism for session review requests using file locks
- Enhance golden check validation with current schema verification
- Add active task tracking and graceful shutdown handling
- Rename check_golden scripts to golden_check for consistency
- Update statistics reporting with correct/incorrect terminology instead of reasonable
- Add stale format detection and compatibility handling for verdict fields
- Include both_correct rate calculation in accuracy metrics
- Add concurrency and staggering options for better resource management
* ci(workflow): add Windows smoke test workflow
- Create new workflow file .github/workflows/windows-smoke.yml
- Configure workflow to trigger on push and pull request events
- Set up Python environment with version 3.11
- Install package dependencies using pip
- Run version job as smoke test for CLI functionality
- Enable concurrency control to prevent duplicate runs
- Use matrix strategy for Python version testing
* feat(benchmark): add retry mechanism and health check for session review
- Added retry configuration options (retry_initial_seconds, retry_max_seconds, retry_max_attempts) to jinli_lme.yaml
- Implemented exponential backoff retry logic with configurable parameters in session_review step
- Added output_is_healthy function to verify session_review.json integrity and absence of failed reviews
- Updated resume functionality to skip only healthy outputs instead of all existing files
- Integrated JSON parsing and validation to check for failed reviews in output files
- Enhanced error handling and logging for retry attempts and recovery scenarios
* feat(benchmark): add LongMemEval session review statistics script
- Create stats_session_review.py to summarize session_review.json artifacts
- Add command line options for listing failed, missing, and run failed samples
- Implement JSON output mode for programmatic consumption
- Calculate and display health statistics including total samples, healthy outputs, failed sessions
- Provide detailed failure information with session IDs and error messages
- Generate re-run commands for samples with failed reviews
- Add percentage calculations for better statistical overview
- Include support for multiple output formats and detailed logging
* feat(benchmark): add LongMemEval output cleanup script and enhance golden check retry logic
- Added clean_sample_outputs.py script to remove generated LongMemEval files while preserving source inputs
- Implemented configurable retry mechanism in golden_check.py with exponential backoff strategy
- Added retry parameters (initial/max seconds and max attempts) to control failure recovery behavior
- Integrated asyncio support for asynchronous sleep during retry intervals
- Configured default retry settings in jinli_lme.yaml with 5s initial and 300s maximum intervals
- Preserved core files (query.json, answer.json, session/) while cleaning generated artifacts
* feat(benchmark): add AppleDouble file cleanup to sample output cleaner
- Remove AppleDouble files starting with '._' recursively including under session/
- Add is_under helper function to check if path is inside parent directory
- Track targets in set to avoid duplicate processing
- Include AppleDouble files in cleanup targets when not already covered by existing targets
- Maintain dry-run mode as default behavior with --apply flag for actual deletion
* refactor(benchmark): update LongMemEval sample output cleaning script
- Add time and Iterator imports for enhanced functionality
- Add --progress-every argument to control progress reporting frequency
- Replace is_under function with iter_sample_targets generator
- Implement detailed progress tracking with timing measurements
- Add sample-by-sample processing with elapsed time reporting
- Include AppleDouble file detection within session directory
- Update target counting and deletion statistics display
- Add conditional progress updates based on progress-every setting
- Improve dry-run mode with would-delete indication
* chore(benchmark): increase initial interval for session review step
- Changed START_INTERVAL_SECONDS from 1.0 to 3.0 seconds
- Adjusted timing parameters for better benchmark stability
* refactor(benchmark): implement coordinated retry mechanism for session reviews
- Add retry gate condition to coordinate concurrent review attempts
- Implement wait_for_healthy_start_slot to handle sequential retries
- Create mark_retrying and mark_recovered functions to track retry states
- Update reply_with_retry to accept index parameter for coordination
- Add has_prior_retry logic to prevent race conditions during recovery
- Ensure proper cleanup of retry state on success or failure
- Maintain backward compatibility while adding coordination features
* chore(benchmark): adjust session review start interval timeout
- Changed START_INTERVAL_SECONDS from 3.0 to 5.0 seconds
- Increased initial delay for session review benchmark step
- Updated timeout configuration for improved stability
* refactor(benchmark): update session review concurrency and throttling mechanism
- Replace global throttle with per-process concurrency control
- Add concurrency parameter with default value of 30 in config
- Add start_interval_seconds parameter with default value of 2 seconds
- Change default concurrency from 3 to 1 in command line interface
- Update documentation to reflect new throttling behavior
- Implement semaphore-based concurrency limiting for review tasks
- Modify retry mechanism to use local locking instead of global files
- Remove portalocker dependency for cross-process throttling
* refactor(config): update session review configuration and concurrency settings
- Removed deprecated retry configuration parameters from jinli_lme.yaml
- Increased MAX_CONCURRENCY from 30 to 60 in session_review.py
- Reduced START_INTERVAL_SECONDS from 2.0 to 1.0 in session_review.py
- Cleaned up redundant backend specifications in configuration file
- Simplified agent wrapper configurations by removing obsolete retry settings
* feat(benchmark): enhance LME auto memory step with advanced scheduling and error handling
- Add datetime parsing functionality for LongMemEval timestamps with regex pattern
- Implement configurable concurrency limits with MAX_CONCURRENCY of 60
- Introduce retry mechanism with exponential backoff for agent interactions
- Add session filtering based on date comparison with question_date validation
- Create rate limiting with start interval control between requests
- Implement sophisticated retry coordination using asyncio conditions
- Add comprehensive error tracking for failed and filtered session extracts
- Remove deprecated concurrency parameter from jinli_lme.yaml configuration
- Add structured output validation in session review step
- Include detailed metadata reporting with session statistics and errors
* fix(benchmark): adjust default concurrency for auto_memory job
- Changed default concurrency from 3 to 1 for auto_memory job to prevent API overload
- Updated help text to reflect new default value of 1 for concurrency parameter
- Modified documentation to clarify concurrency behavior varies by job type
* refactor(search): replace hardcoded candidate multiplier with constant
- Introduced _CANDIDATE_MULTIPLIER constant set to 10
- Replaced hardcoded factor of 5 with _CANDIDATE_MULTIPLIER in BM25 search
- Replaced hardcoded factor of 5 with _CANDIDATE_MULTIPLIER in vector search
- Updated test to verify both search steps use ten times limit for candidates
- Imported VectorSearchStep and Bm25SearchStep in test module
- Added comprehensive test case for candidate count calculation logic
* feat(lme): add data inspection error handling with fallback mechanism
- Implemented non-retryable data inspection error markers detection
- Added _is_data_inspection_error method to identify inspection failures
- Created fallback handling for data inspection errors in auto memory extraction
- Added fallback handling for data inspection errors in session review
- Extended failed extracts tracking with non-retryable and fallback flags
- Separated fallback extracts from regular failed extracts in reporting
- Enhanced error logging with specific data inspection failure messages
- Updated metrics to track fallback extractions and reviews separately
- Maintained existing retry logic for other exception types
* feat(benchmark): enhance session review statistics with fallback tracking
- Add support for identifying and listing non-retryable fallback reviews
- Introduce --list-fallback argument to display fallback review details
- Separate retryable failures from non-retryable fallbacks in reporting
- Track fallback samples and sessions separately from failed ones
- Update console output to show both retryable and non-retryable categories
- Include fallback details in JSON output with reasons and session info
- Modify failure counting logic to distinguish between retryable and fallback reviews
* feat(benchmark): add question_id tracking and enhanced fallback reporting
- Add question_id function to extract query.question_id from data
- Initialize question_id_by_id dictionary to store question IDs by index
- Store question_id for each sample during data processing
- Enhance fallback output to include question IDs and session information
- Format sample labels with question IDs when available
- Display session IDs associated with each fallback case
* feat(benchmark): add question_id support and improve bad sample reporting
- Add question_id_for function to extract question_id from multiple sources
- Add sample_label function to format samples as idx(question_id) when available
- Store question_id in data dictionary during processing
- Change bad_golden and bad_sessions to store full records instead of just indices
- Update list_bad output to show formatted labels with question_id information
- Improve error reporting with more detailed sample identification
* feat(benchmark): enhance golden check stats with structured output
- Add related_session_ids function to extract session IDs from verdict records
- Create grouped_records function to group records by question type
- Replace flat list output with JSON-formatted grouped records in list_bad option
- Replace flat list output with JSON-formatted grouped records in list_bad_sessions option
- Maintain Chinese labels while adding structured data presentation
- Improve readability of bad verdict record display with hierarchical grouping
* feat(benchmark): update data structure for question indexing
- Replace sample_label with _idx field for index tracking
- Add question_id field to store _question_id values
- Maintain backward compatibility with empty string defaults
- Preserve existing session_id functionality
- Update data mapping to include new fields in grouped results
* refactor(benchmark): streamline golden answer verification process
- Replace relevance filtering with comprehensive information extraction
- Remove is_relevant field and simplify session summary structure
- Change relevant_info to extracted_info for clarity
- Update golden check logic to work with full extractions instead of filtered summaries
- Simplify prompt instructions to focus on complete information extraction
- Remove redundant schema validation and structured output requirements
- Adjust statistics calculation to match new extraction approach
- Update metadata field names to reflect extraction rather than relevance checking
* feat(benchmark): add selective file deletion option to clean_sample_outputs
- Add --filename argument to delete only specific root-level files
- Modify iter_sample_targets function to accept optional filenames filter
- Implement validation for root-level filename constraints
- Update function calls to pass filenames parameter
- Add example usage for selective file deletion in documentation
* feat(benchmark): add error count metrics to golden check statistics
- Added golden_bad, session_bad, and both_bad calculation fields
- Updated console output format to include error counts per question type
- Modified table display to show both accuracy rates and error numbers
- Enhanced statistical summary with additional error breakdown metrics
* test(search): update search step tests with include_source parameter
- Added include_source=False parameter to VectorSearchStep initialization
- Added include_source=False parameter to Bm25SearchStep initialization
- Maintained existing RuntimeContext parameters for both search steps
- Updated test calls to match new constructor signature with include_source option
* refactor(embedding): update embedding model initialization and session storage paths
- Remove unused inspect import from as_embedding module
- Pass dimensions directly to embedding model constructor instead of using parameters
- Update session state file paths to use mem_session directory instead of resource
- Add mem_session_dir configuration option to application config schema
- Update workspace directory creation to include new mem_session directory
- Change AgentScope and Claude Code session paths to use mem_session directory
- Move embedding dimensions from parameters to top-level configuration
- Update AgentScope dependency version from 2.0.3 to 2.0.4
- Update integration tests to reflect new session file location paths
* chore(version): bump version to 0.4.0.8
- Update __version__ from 0.4.0.7 to 0.4.0.8 in __init__.py
* refactor(transfer): drop orphaned ingest step, make service discovery cross-platform
- Remove ingest step: superseded by auto_resource (drop files under
resource/ → watcher interprets them); its meta.json/<date>.md outputs
had no consumers and tripped the auto_resource watcher.
- Replace lsof/pgrep shell-outs in service_utils with psutil (per-process
enumeration, no root needed on macOS) for Windows/macOS/Linux support.
- Add cross-platform test coverage for _pid_on_port / _scan_reme_procs.
- Deps: +psutil, -filelock (only used by the removed ingest lock).
* chore(release): bump version to 0.4.0.5
* fix(mcp): resolve circular import issues and update dependencies
- Moved fastmcp imports inside functions to prevent circular dependencies
- Replaced _TRANSPORT_MAP with _VALID_TRANSPORTS set for transport validation
- Updated version number from 0.4.0.3 to 0.4.0.4
- Added claude-agent-sdk dependency to core optional dependencies
- Used TYPE_CHECKING imports for FastMCP related types
- Restructured transport mapping logic within function scope
- Fixed string annotation for CallToolResult type hints
* refactor(tests): update date handling in daily steps tests
- Replace _date.today() with timezone-aware now function
- Use Asia/Shanghai timezone for date formatting
- Change return format to use strftime instead of isoformat
- Import now function from reme.steps.evolve module
* refactor(tests): clean up unused imports in daily steps test
- Removed unused date import from datetime module
- Removed redundant pathlib Path import that was already imported later
- Kept necessary imports for asyncio, os, tempfile, warnings, and frontmatter modules
* test(daily_steps): update test to include application context for daily list step
- Add ApplicationContext initialization with temporary workspace directory
- Register file store component in application context
- Pass application context to DailyListStep constructor
- Maintain existing test assertion behavior for date metadata verification
- Changed project name in pyproject.toml from 'reme' to 'reme-ai'
- Updated dependency references in full extras to use 'reme-ai[core]' and 'reme-ai[dev]'
* refactor(file_chunker): replace file parser with file chunker component
- Rename file_parser module to file_chunker across codebase
- Update BaseFileParser to BaseFileChunker with corresponding component type
- Rename LinkedFileParser to MarkdownFileChunker for markdown-specific chunking
- Rename ChunkedFileParser to DefaultFileChunker for default byte-based chunking
- Update documentation references from file_parser to file_chunker
- Modify dependency injection in BaseStep to use file_chunker instead of file_parser
- Update configuration and component registration to use new chunker naming
- Rename all related test files and update test assertions accordingly
- Add recursive option to scan_store_changes_step in default configuration
* feat(database): enhance Neo4j connection with environment variable support
- Add support for NEO4J_PASSWORD environment variable as fallback
- Make password parameter optional in constructor with validation
- Update chromadb dependency from 1.3.5 to 1.5.7
- Configure CORS credentials based on origin settings
- Import os module for environment variable access
* feat(config): add timezone support and remove unused dialog directory
- Added timezone field to application config with IANA timezone support
- Removed unused dialog_dir configuration and related directory creation
- Replaced date.today() with timezone-aware now() function across daily operations
- Created evolve module with timezone-aware datetime functionality
- Updated daily_create, daily_list, and daily_reindex steps to use timezone-aware dates
* refactor(steps): update file chunker implementation
- Replace ChunkedFileParser with DefaultFileChunker in background steps
- Add module docstring to evolve steps package
- Update return type annotation to reflect new chunker class usage
* refactor(components): rename embedding and llm components to as_embedding and as_llm
- Rename reme4/components/embedding to reme4/components/as_embedding
- Rename reme4/components/llm to reme4/components/as_llm
- Update all imports and references from embedding to as_embedding
- Update all imports and references from llm to as_llm
- Change BaseEmbedding to BaseAsEmbedding and update inheritance
- Change BaseLLM to BaseAsLLM and update inheritance
- Update component types from LLM/EMBEDDING to AS_LLM/AS_EMBEDDING
- Update configuration keys from embedding/llm to as_embedding/as_llm
- Update all property references from llm to as_llm in step classes
- Update test assertions to use new component enum values
* refactor(embedding_store): rename embedding parameter to as_embedding
- Updated configuration key from 'embedding' to 'as_embedding'
- Renamed class attribute from 'embedding' to 'as_embedding'
- Updated method calls to use 'as_embedding' instead of 'embedding'
- Changed parameter name in constructor from 'embedding' to 'as_embedding'
- Updated documentation to reflect new parameter name
- Modified health check to use 'as_embedding' property
* feat(agent_wrapper): add unified agent wrapper component with multiple backends
- Introduce BaseAgentWrapper abstract base class for agent implementations
- Add AsAgentWrapper implementation using AgentScope framework
- Add CcAgentWrapper implementation using Claude Code SDK
- Register agent_wrapper component type in ComponentEnum
- Configure default agent_wrapper settings in default.yaml
- Implement tool integration for both AgentScope and Claude Code backends
- Support fluent configuration via set_system_prompt() and add_tools() methods
* feat(agent-wrapper): add structured output support for agent wrappers
- Import SystemMsg in AsAgentWrapper for structured output handling
- Add output_schema parameter support in AsAgentWrapper with generate_structured_output
- Implement set_output_schema method in BaseAgentWrapper for chaining configuration
- Add output schema support in CcAgentWrapper with JSON schema format option
- Return structured output when available in CcAgentWrapper response
- Refactor kwargs handling to use default values consistently across wrapper classes
* refactor(vector_store): make obvec and zvec vector stores optional dependencies
- Removed direct imports of ObVecVectorStore and ZvecVectorStore from init file
- Added try-except blocks for conditional importing of optional vector stores
- Updated error handling to check for both pyobvector and sqlalchemy in ObVecVectorStore
- Renamed _OBVECTOR_IMPORT_ERROR to _OBVEC_IMPORT_ERROR for consistency
- Moved pyobvector and related dependencies to optional 'obvec' extra
- Added separate 'zvec' optional dependency group
- Updated package configuration to exclude reme4 module patterns
- Removed reme4 entry point from console scripts
- Bumped version from 0.3.1.9 to 0.3.1.10
* refactor(dependencies): reorganize project dependencies and add optional seekdb support
- Move sqlite-vec, prompt_toolkit, and rich to earlier in dependencies list
- Remove pyseekdb from main dependencies and create separate seekdb optional dependency group
- Reorder pyyaml to later in the dependencies list
- Maintain all existing dependency versions while improving organization
* chore(deps): remove faiss-cpu dependency from pyproject.toml
- Removed faiss-cpu>=1.7.4 from the faiss dependency group
- Cleaned up unused faiss dependency configuration
- Updated project dependencies to exclude faiss-cpu package
* refactor(evolve): consolidate auto memory planner and writer into single step
- Removed separate AutoMemoryPlannerStep and AutoMemoryWriterStep classes
- Combined functionality into new AutoMemoryStep class in auto_memory.py
- Migrated prompt templates from separate YAML files to unified auto_memory.yaml
- Updated module imports to reference new consolidated step
- Simplified memory recording process using single ReAct agent instead of two-stage planning/writing
- Maintained same input/output contract with messages, session_id, and memory_hint parameters
- Preserved all original functionality for creating/updating daily notes with conversation facts
* fix(daily): update empty session_id handling to create day-level file
- Changed test to verify empty session_id creates day-level file daily/<date>.md
- Updated assertion to check response success instead of rejection
- Modified metadata verification to include path, session_id and created status
- Added file existence check for the generated daily markdown file
- Updated test name and print statement to reflect new behavior
- Fixed test registration to use updated function name
* fix(bm25_index): 修正BM25索引计算中的文档长度归一化问题
修复了在计算BM25相似度时对文档长度进行不正确归一化的bug，确保所有查询都能得到准确的相关性评分。
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* refactor(steps): Rename and adjust indexing step logic
- Rename `scan_changes.py` and `reindex.py` to `clear_and_scan.py`
- Update implementation details of `ScanChangesStep` and `ClearAndScanStep`
- Modify the scheduling mechanism in `WatchChangesStep`
- Adjust step registration and parameter configuration in config files
- Update related tests to align with the new interface changes
* up
* feat(daily): replace daily CRUD operations with slug provisioning approach
* refactor(tests): migrate CRUD step tests from HTTP server to direct LocalFileStore
* up
* up
* up
* up
---------
Co-authored-by: huangsen <huangsen.huang@alibaba-inc.com>
* feat(file_store): add FAISS-backed local file store implementation
- Introduce FaissLocalFileStore class with vector search capabilities using FAISS IndexFlatIP
- Implement FAISS index persistence with binary format and JSON id-map sidecar
- Add automatic index rebuilding when sidecar files are missing or corrupted
- Support tombstone mechanism for efficient deletion and compaction
- Register 'faiss' component type in the registry system
- Add faiss-cpu dependency requirement to pyproject.toml
- Update configuration schema to use simplified parameter structure
- Enhance search step to support parameter override from runtime context
- Add comprehensive unit tests for FAISS store functionality
- Implement fallback to parent methods for basic CRUD operations
* refactor(search): simplify parameter retrieval logic
- Removed _param method that checked context and kwargs
- Directly use self.kwargs.get for all parameter retrievals
- Maintained same default values for vector_weight, candidate_multiplier, expand_links, and max_links_per_direction
- Reduced code complexity by eliminating redundant context checking logic
* feat(seekdb): add Seekdb file and vector stores with pyseekdb>=1.2.0
* refactor(seekdb): add pyseekdb_conn and remote-only host/port config
* refactor(embedding): remove env fallbacks from BaseEmbeddingModel; pass credentials in tests
* refactor(seekdb): drop tenant from client kwargs; default database test and empty password
* fix(deps): gate pyseekdb to Python >=3.11 for CI 3.10 compatibility
* fix(seekdb): satisfy pre-commit pylint and formatting for seekdb stores
* refactor(steps): Add job management methods and support registering them as tools
Added methods to the `BaseStep` class for retrieving, running, and registering jobs as tools, enhancing the functionality of the step class.
* fix doc
* chore(pyproject.toml): Update dependency versions and adjust package configuration
Bump agentscope version to 1.0.19 and reorganize the core dependency configuration structure.
* feat: add Neo4j file graph support and markdown parser with wikilink extraction
- Add Neo4jFileGraph implementation for property-graph storage with
virtual/real node handling and link management
- Introduce LinkedFileParser for markdown files with frontmatter,
wikilink graph extraction, and full-skeleton chunking
- Update pyproject.toml to include pyyaml, mistletoe, and neo4j
dependencies
- Modify .gitignore to exclude /vault and structure.md
- Change reme CLI entry point from reme_ai.main to remecli.reme
- Register new neo4j and md components in respective registries
* refactor(file-graph): add chunk_ids support to Neo4jFileGraph
Add chunk_ids field to File node properties in Neo4jFileGraph to
enable better content chunk tracking and management.
BREAKING CHANGE: File node schema now includes chunk_ids property
which may affect existing integrations.
feat(parser): implement wikilink resolution logic
Move path resolution logic from utils/path_resolver to
linked_file_parser module and enhance wikilink resolution with
folder-note rule support and improved error handling.
fix(tests): update test assertions and variable names
Update test cases to reflect changes in data structures and
variable naming conventions across various components.
chore(config): update package entry point reference
Change reme CLI entry point from remecli.reme:main to
reme_ai.reme:main in pyproject.toml.
refactor(utils): remove deprecated path_resolver module
Remove the old path_resolver utility module as its functionality
has been moved to linked_file_parser.
docs(file-graph): update Neo4jFileGraph documentation
Update class docstrings and comments to reflect new chunk_ids
property and other structural changes.
style(formatting): adjust code formatting and line breaks
Minor formatting improvements including line length optimization
and consistent spacing adjustments throughout the codebase.
* fix(pyproject.toml): correct entry point for reme command
Change the entry point from "reme_ai.reme:main" to "reme_ai.main:main"
to fix the module reference for the reme command in project scripts.
* feat(vector_store): add OceanBase as a VectorStore
* refactor(obvec): make it cleaner
* docs: add obvec related info
* refactor: minor update
* refactor: clean code and pass lint
* docs: remove unrelated edit
* docs: minor update
* feat(compactor): add extra instruction support and improve error handling
- Add extra_instruction parameter to compactor for custom guidance during message compaction
- Implement try-catch blocks around AS LLM initialization with detailed error logging
- Add extra_instruction parameter to ReMe.compact method with comprehensive documentation
- Update agentscope dependency from 1.0.17 to 1.0.18 in light installation
- Bump version number from 0.3.1.6 to 0.3.1.7
- Pass extra_instruction parameter through compactor instantiation and execution flow
* fix(core): add error handling for AS LLM formatters and token counters initialization
- Wrapped AS LLM formatters initialization in try-except blocks
- Added specific error logging for failed AS LLM formatter initialization
- Wrapped AS token counters initialization in try-except blocks
- Added specific error logging for failed AS token counter initialization
- Applied same error handling pattern to both initial setup and restart operations
- Maintained existing warning logs for unsupported backends
* refactor(core): replace text truncation utilities with new marker system
- Remove old truncate_text_utils module and its exports
- Replace TRUNCATION_MARKER_START with _TRUNCATION_NOTICE_MARKER constant
- Update as_msg_stat.py to split content using new marker format
- Modify FileIO tool to use TRUNCATION_NOTICE_MARKER for continuation hints
- Change is_truncated function checks to use marker presence detection
- Move transformers dependency from main deps to light extra dependencies
- Update tool result compactor tests to verify marker instead of is_truncated calls
* feat(file_io): enhance file operations with path resolution and append functionality
- Add expanduser() to resolve file paths with ~ symbol
- Implement proper file existence and type validation in update_file
- Add new append_file method to append content to files
- Update truncation notice format for better readability
- Fix typo in error message from "provide" to "provided"
- Update transformers dependency in pyproject.toml
- Remove duplicate transformers dependency from light extras
* refactor(file_io): disable pylint too-many-return-statements warning
* perf(file_watcher): increase default polling delay and optimize watcher configuration
- Increased default poll_delay_ms from 1000ms to 2000ms to reduce CPU usage
- Removed force_polling parameter as it's no longer needed with updated polling strategy
- Simplified async watch configuration by removing conditional force_polling logic
- Reduced overall system resource consumption during file watching operations
* refactor(memory): update conversation log documentation in memory summary
- Changed "Raw conversation logs" to "Earlier conversation logs" for clarity
- Added warning note about potentially large dialog file sizes
- Improved formatting with additional line break for better readability
- Maintained existing compressed summary integration unchanged
* feat(memory): add long-term memory support to file-based memory system
- Initialize _long_term_memory attribute as empty string
- Add memories section to content when long-term memory exists
- Consolidate summary and memories into single user message
- Format memories with markdown header # Memories
- Maintain existing compressed summary functionality
- Join multiple content parts with double newlines
* chore(deps): update version and move litellm to dev dependencies
- Updated package version from 0.3.1.2 to 0.3.1.3
- Removed litellm from main dependencies in pyproject.toml
- Added litellm as fixed version dependency in dev group
- Maintained litellm requirement while reorganizing dependency structure
* chore(deps): move litellm dependency to full extras
- Moved litellm==1.80.0 from main dependencies to full extra
- Kept litellm as optional dependency for users needing full feature set
- Maintains backward compatibility for light installation option
* feat(pyproject): add litellm dependency to project configuration
- Added litellm==1.80.0 as optional dependency in pyproject.toml
- Created new litellm extra group for LiteLLM integration
- Updated full dependency group to include the new litellm option
* update
* refactor(memory): remove unnecessary type check and update error logging
* refactor(core): standardize logger import and update agentscope dependency
* fix(memory): disable console output and add logging for summarizer component
* feat(core): replace OpenAI token counter with custom ReMe token counter
- Replace OpenAITokenCounter with ReMeTokenCounter implementation
- Add support for HuggingFace mirror and configurable tokenizer
- Register ReMeTokenCounter as default token counter in registry
- Update config to use hf backend with Qwen2.5-7B-Instruct model
refactor(memory): convert token counting methods to async in message handlers
- Change count_str_token, stat_message, count_msgs_token to async methods
- Update format_msgs_to_str and context_check to use async token counting
- Modify _format_tool_result_output to support async token counting
- Adjust all dependent methods to await async token counting calls
feat(memory): add dialog persistence to in-memory storage
- Implement _append_messages_to_dialog for saving messages to JSONL files
- Add dialog_path parameter to ReMeInMemoryMemory constructor
- Persist messages to daily JSONL files based on timestamp grouping
- Update mark_messages_compressed to save and remove compressed messages
- Modify clear_content to persist all messages before clearing memory
refactor(ops): update token counter type hints and initialization
- Change BaseOp to use HuggingFaceTokenCounter instead of TokenCounterBase
- Update type annotations for as_token_counter property and parameters
- Remove direct token counter injection from Compactor and ContextChecker
- Pass as_token_counter parameter through service context mechanism
style(logging): improve error logging with exception details
- Replace logger.error with logger.exception in browser control tool
- Change logger.error to logger.exception in memory get tool error handling
- Add proper exception logging with stack trace information
chore(config): add token counter configuration to light YAML
- Add as_token_counters section with default hf backend configuration
- Configure Qwen/Qwen2.5-7B-Instruct model with mirror support enabled
- Set up pretrained_model_name_or_path and use_mirror parameters
test(context): update context check tests to async implementation
- Convert verify_context_check_invariants to async function
- Update context check test methods to use async calls
- Change stat_message calls to await async implementation
- Modify test_empty_messages and test_below_threshold_returns_all to async
* feat(core): implement context checking and memory management features
* refactor(core): replace direct loguru import with logger utility function
* refactor(reme): remove RuntimeContext dependency and simplify context checking
* feat(docs): add raw conversation persistence to ReMe framework
- Add Weikang Zhou as a contributor in pyproject.toml
- Update README.md with new author in software citation
- Add new paper reference for AgentscopeReMe framework
- Include arXiv link and publication details
- Add full author list for the research paper
- Update bibliography with proper formatting
