* test(background_steps): add comprehensive tests for batch processing and memory management
- Add test for catalog upserts in batches of at most 100 files
- Add test for catalog deletes in batches of at most 100 paths
- Add test for index memory budget reducing batches to one file
- Add test for memory target limiting cumulative batch size
- Add test for invalid batch memory settings rejection
- Add test for continuing after one batch fails
- Add test for yielding to event loop while building batch
- Add test for modified file reusing unchanged embedding
- Add test for reporting memory estimation failure without aborting
feat(update_changes): implement bounded batch processing with memory management
- Add configurable batch parameters with default values
- Implement memory budget calculation based on available system memory
- Add file inspection and memory estimation before processing
- Implement batch flushing when limits are reached
- Add proper error handling for batch operations
- Support async yielding during batch building
- Add comprehensive validation for batch configuration parameters
- Implement memory estimation for indexing operations
- Add batch size limiting for delete operations
* test(steps): add tests for memory estimation failure handling
- Add test case for isolated file processing when memory estimation fails
- Add test case for proper release of flushed items before building next file
- Implement weak reference tracking to verify payload lifetime management
- Create parametrized tests for both source and item memory estimation methods
- Add assertions to verify single-item batch behavior on estimation failures
- Include comprehensive error handling verification for memory budget calculations
* chore(version): bump version to 0.4.1.3
- Update __version__ from 0.4.1.2 to 0.4.1.3 in __init__.py
* feat(index): support batch settings from environment
* refactor(index): use direct batch defaults
* refactor(index): configure memory estimates through step args
* ci: simplify Windows smoke dependencies
* feat(codex): add authentication mode support with thread-safe logging
- Implement _CodexAuthConfig dataclass for resolved auth settings
- Add auth_mode parameter with auto/api_key/oauth options
- Separate API key and OAuth authentication flows
- Force specific login method based on auth mode
- Add explicit API key validation requirement
- Serialize concurrent logger initialization in thread lock
- Close logging handlers properly during cleanup
- Update default config with auth_mode presets for codex and codex_oauth
- Add comprehensive tests for authentication modes and concurrent logging
* feat(file_store): implement background embedding backfill and keyword index repair
- Add _after_embedding_backfill hook in FAISS local file store
- Schedule startup embedding repair without delaying component readiness
- Cancel and collect embedding backfill task during component shutdown
- Log progress at fixed percentage boundaries for long-running operations
- Process embedding backfill in configurable batch sizes with progress reporting
- Rebuild keyword index in bounded batches with detailed mismatch diagnostics
- Format stdlib logs consistently with QwenPaw console output using relative paths
- Run embedding backfill as background task that doesn't block component startup
- Add comprehensive tests for background embedding and keyword index repair scenarios
* fix(file-store): repair graph-chunk consistency on load
- Add _repair_graph_chunk_consistency method to detect and fix mismatched graph/chunk states
- Clear torn graph/chunk state when missing or orphaned chunks are detected
- Ensure keyword index sync handles empty chunks properly
- Add comprehensive tests for graph-chunk consistency scenarios
- Update test utilities to properly seed graph/chunk snapshots
- Increment version to 0.4.1.2
* feat(file_io): enhance list step response format and add comprehensive logging
- Format list output with bullet points for better readability
- Add explicit "No files found" message when directory is empty
- Include detailed timing information for file store startup phases
- Add logging for chunk loading, graph consistency checks, and keyword indexing
- Provide detailed metrics for embedding backfill operations
- Add comprehensive test coverage for empty directory scenarios
- Include batch processing statistics for embedding operations
* feat(logger): add QwenPaw logging integration with forwarding mechanism
- Introduce _ForwardToLoggerHandler to forward log records to target logger
- Add qwenpaw logger integration that forwards ReMe logs to QwenPaw handlers
- Maintain ReMe logger stability for modules that cache it at import time
- Enable QwenPaw handlers to take effect without ReMe reconfiguration
- Add comprehensive tests for stdlib forwarding to QwenPaw sinks
- Support explicit REME_DISABLE_LOGURU=false to keep original Loguru backend
- Preserve existing logging behavior when QwenPaw is not configured
* fix(file_store): serialize concurrent FAISS dump operations to prevent corruption
- Add asyncio lock to ensure only one FAISS dump operation runs at a time
- Generate unique temporary filenames using UUID tokens for atomic replacement
- Implement proper cleanup of temporary files in finally block
- Add comprehensive test to verify concurrent dumps are serialized
- Ensure atomic writes by replacing both index and idmap files together
- Prevent partial state writes during concurrent access scenarios
- Removed max_file_bytes configuration from index_update_loop, resource_watch_loop, digest_watch_loop, and reindex jobs
- Updated default.yaml to reflect simplified job configurations without file size limits
- Removed corresponding test case that validated the 20 MiB limit behavior
- Simplified watch directories and suffixes to basic configurations
- Cleaned up unnecessary commented configurations in the YAML file
* feat(benchmark): add final answer review step for evaluation
- Introduce FinalAnswerReviewStep to handle answer validation
- Add final_answer_review.jsonl dataset with 24 evaluation cases
- Include detailed reasoning and golden check results for each case
- Support various question types including temporal reasoning and preferences
- Implement time consistency checks for session references
- Add comprehensive test coverage for different evaluation scenarios
* chore(benchmark): remove longmemeval final answer review file
- Removed final_answer_review.jsonl containing 23 evaluation records
- Deleted question_id mappings with detailed reasoning for golden answers
- Removed answer correctness assessments and session time validation checks
- Cleaned up benchmark dataset used for memory evaluation testing
- Eliminated JSONL format evaluation results for temporal reasoning tasks
- Removed references to various session IDs and time-based validations
* config(default): disable shell step configuration by commenting out
- Commented out the shell step configuration in default.yaml
- Disabled asynchronous shell command execution capability
- Removed shell step from available backend operations
- Preserved traverse backend configuration unchanged
* refactor(tests): remove unused shell job test from config parser tests
- Removed test_default_config_registers_shell_job function that was no longer needed
- Kept existing test for frontmatter chunk metadata configuration
- Cleaned up test suite by removing obsolete test case
* feat(index): add file size limits and oversized file handling
- Implement max_file_bytes configuration option for content processing jobs
- Add default 20MB file size limit for background processing in default config
- Skip oversized files during auto_resource step with appropriate metadata
- Clear stale index entries when oversized files are modified
- Add size-based filtering logic to update_changes step with skip reporting
- Include file size validation in UpdateIndexStep with proper response handling
- Add comprehensive tests for oversized file scenarios in auto_resource and update_index
- Document file size limits in constants with appropriate thresholds
* chore(version): bump version to 0.4.1.1
- Update __version__ from 0.4.1.0 to 0.4.1.1 in __init__.py
* fix(index): isolate batch metadata and handle file races
- Changed dimensions property to avoid forcing provider construction
- Added _ensure_model method to construct provider on demand
- Modified __call__ to ensure model exists before use
- Updated _start to defer provider initialization
- Removed eager health check during startup
- Added compact embedding serialization with base64 encoding
- Implemented batch processing for vector search with heap-based ranking
- Added document_ids property to keyword index interface
- Updated chunk persistence to handle legacy JSON embeddings
- Optimized memory usage by avoiding materialization of metadata in document_ids
* feat(core): add shell command execution and memory status reporting
- Introduce ShellStep for executing shell commands with timeout support
- Add StatusStep to report memory estimates for stateful data components
- Register shell and status commands in default configuration
- Update documentation with new reme status and shell command capabilities
- Implement comprehensive unit tests for both new step types
- Add support for asynchronous command execution with proper error handling
* feat(config): add log_config option to suppress config loading logs
- Add log_config parameter to resolve_app_config function with default True
- Conditionally log config loading messages based on log_config flag
- Update reme.py and service_utils.py to use log_config=False for client calls
- Suppress config logging in user-facing contexts to avoid output pollution
refactor(shell): rename command parameter to cmd for clarity
- Change 'command' to 'cmd' in default.yaml configuration schema
- Rename 'timeout' to 'shell_timeout' to avoid parameter name collisions
- Update ShellStep to accept both legacy and new parameter names
- Maintain backward compatibility with existing command/timeout usage
test(shell): add comprehensive tests for shell step parameter handling
- Add test cases for new cmd and shell_timeout parameter names
- Verify legacy command and timeout parameters still work
- Test blank command rejection message updated to use cmd
- Create integration test for shell parameter payload passing
* fix(shell): ensure proper environment loading and process timeout handling
- Move load_env() call to execute before parse_args() in main function
- Add proper process group killing for timeout scenarios on POSIX systems
- Implement recursive child process termination on Windows for proper cleanup
- Change parameter name from 'timeout' to 'shell_timeout' in shell execution
- Remove support for legacy 'command' and 'timeout' parameter names
- Update test cases to verify new timeout behavior and parameter requirements
- Add comments explaining component size tracking implementation details
* feat(benchmark): add golden answer validation and session review for LongMemEval
- Introduce GoldenCheckStep to validate LongMemEval golden answers using structured verdicts
- Add SessionReviewStep to extract query/answer-relevant evidence from all sessions
- Implement concurrent session processing with configurable concurrency limits
- Create check_golden job configuration with lme_review and lme_judge agent wrappers
- Add Qwen3.7-plus model configuration for enhanced processing capabilities
- Include python_execute tool integration for agent-based reasoning and date validation
- Generate comprehensive JSON output with session summaries and validation verdicts
- Add run_check_golden.py script for batch processing across all LongMemEval samples
- Configure proper logging initialization with console and file output options
- Update component registry and file I/O modules to support new benchmark features
* feat(scripts): add script to summarize LongMemEval check_golden verdicts
- Parse check_golden.json files across all LongMemEval samples
- Calculate accuracy metrics for golden answers and session IDs
- Provide breakdown by question type with percentage calculations
- Add command line options for listing bad samples and JSON output
- Include progress tracking showing completed vs pending samples
- Display confidence scores and date sanity checks statistics
* refactor(benchmark): move golden check scripts to longmemeval directory
- Moved run_check_golden.py from scripts/ to benchmark/longmemeval/
- Moved stats_check_golden.py from scripts/ to benchmark/longmemeval/
- Updated path resolution to use parents[2] instead of parent.parent
- Added new --list-run-failed option to stats script
- Added logging directory constant and functions for tracking launched samples
- Enhanced stats output with launched count and run failure information
- Improved error reporting with run failure details and log file paths
* feat(benchmark): add LongMemEval agentic answer workflow with session extraction
- Add LmeAgenticAnswerStep, LmeAutoMemoryStep, and LmeExtractSessionStep to __init__.py
- Create shared helper render_with_source for displaying search results with session_id
- Implement agentic_answer step with vector_search, bm25_search, and extract_session_by_id tools
- Add auto_memory step to convert each session into search-friendly daily notes
- Create extract_session step to retrieve and analyze raw session content by session_id
- Update jinli_lme.yaml with auto_memory, vector_search, bm25_search, and agentic_answer jobs
- Configure lme_memory, lme_extract, and lme_agentic_answer agent wrappers
- Enhance search steps with include_source option to show session_id metadata
- Add proper session_id tracking and collision handling in daily note generation
* feat(benchmark): add LongMemEval agentic answer evaluation pipeline
- Add session_id tracking to agentic_answer.py result metadata
- Introduce run_agentic_answer.py driver for complete pipeline execution
- Implement auto_memory, update_index, and agentic_answer job orchestration
- Add concurrent execution with configurable limits and staggering
- Create aggregation script for collecting tool-call trails and results
- Add stats_agentic_answer.py for comprehensive result analysis
- Implement resume capability with existing output detection
- Generate aggregate.json with per-sample breakdown and tool call summaries
* feat(steps): add ClearPathsStep for cleaning workspace outputs before rebuild
- Introduce ClearPathsStep to remove stale workspace files/directories
- Add support for specifying paths and config_keys as targets to clear
- Implement safety checks to prevent deletion of files outside workspace
- Add logging for cleared paths and warnings for invalid paths
- Configure clear_paths_step in jinli_lme.yaml to clean daily_dir
- Add clear_paths_step to clean mem_answer.json before rebuilds
* feat(benchmark): add resume functionality to agentic answer runner
- Replace --force flag with --resume flag for controlling job execution
- By default every job reruns with clean rebuild behavior using config clear steps
- Add --resume option to skip samples whose output already exists and continue interrupted batches
- Update documentation to reflect new default clean rebuild behavior
- Modify job skipping logic to honor resume flag instead of force flag
- Update dry-run output to show correct todo jobs based on resume status
- Change default example command to use --resume for continuing interrupted runs
* feat(benchmark): generate JSONL output for check golden records
- Add write_check_golden_list function to create JSONL file
- Write all readable check_golden records as JSONL format
- Include check_golden_list path in stats output
- Display generated JSONL file path in summary report
- Maintain UTF-8 encoding with non-ASCII character support
* refactor(benchmark): rename answer judge step and integrate LME LLM judge
- Rename AnswerJudgeStep to LmeLlmJudgeStep and update imports
- Add new llm_judge configuration in jinli_lme.yaml
- Update run_agentic_answer.py to include llm_judge in pipeline
- Modify LmeLlmJudgeStep to read from query.json and answer.json
- Write LLM judgement results back to mem_answer.json
- Add command line options for start/end sample range selection
- Update aggregate.json generation to include LLM judgement data
- Add resume capability for llm_judge job based on judgement presence
* refactor(benchmark): rename answer judge step and integrate LME LLM judge
- Rename AnswerJudgeStep to LmeLlmJudgeStep and update imports
- Add new llm_judge configuration in jinli_lme.yaml
- Update run_agentic_answer.py to include llm_judge in pipeline
- Modify LmeLlmJudgeStep to read from query.json and answer.json
- Write LLM judgement results back to mem_answer.json
- Add command line options for start/end sample range selection
- Update aggregate.json generation to include LLM judgement data
- Add resume capability for llm_judge job based on judgement presence
* feat(steps): add wait_for_paths_step to block until workspace files exist
- Introduce WaitForPathsStep class that polls for required workspace-relative paths
- Add step registration with 'wait_for_paths_step' backend identifier
- Implement path validation to ensure targets are within workspace boundaries
- Add polling mechanism with configurable intervals via poll_seconds parameter
- Include logging functionality with log_every_seconds parameter for status updates
- Add metadata tracking of waited paths and duration in response object
- Register step in index module and expose in public API
- Configure step in jinli_lme.yaml to wait for session_review.json before golden check
- Add script rename from run_check_golden.py to run_golden_check.py with enhanced options
* feat(benchmark): enhance longmemeval benchmarking with concurrency and progress tracking
- Add benchmark extra dependency group with portalocker requirement
- Introduce concurrent execution support for golden_check and session_review workflows
- Add progress reporting interval option with real-time status updates
- Implement global throttling mechanism for session review requests using file locks
- Enhance golden check validation with current schema verification
- Add active task tracking and graceful shutdown handling
- Rename check_golden scripts to golden_check for consistency
- Update statistics reporting with correct/incorrect terminology instead of reasonable
- Add stale format detection and compatibility handling for verdict fields
- Include both_correct rate calculation in accuracy metrics
- Add concurrency and staggering options for better resource management
* ci(workflow): add Windows smoke test workflow
- Create new workflow file .github/workflows/windows-smoke.yml
- Configure workflow to trigger on push and pull request events
- Set up Python environment with version 3.11
- Install package dependencies using pip
- Run version job as smoke test for CLI functionality
- Enable concurrency control to prevent duplicate runs
- Use matrix strategy for Python version testing
* feat(benchmark): add retry mechanism and health check for session review
- Added retry configuration options (retry_initial_seconds, retry_max_seconds, retry_max_attempts) to jinli_lme.yaml
- Implemented exponential backoff retry logic with configurable parameters in session_review step
- Added output_is_healthy function to verify session_review.json integrity and absence of failed reviews
- Updated resume functionality to skip only healthy outputs instead of all existing files
- Integrated JSON parsing and validation to check for failed reviews in output files
- Enhanced error handling and logging for retry attempts and recovery scenarios
* feat(benchmark): add LongMemEval session review statistics script
- Create stats_session_review.py to summarize session_review.json artifacts
- Add command line options for listing failed, missing, and run failed samples
- Implement JSON output mode for programmatic consumption
- Calculate and display health statistics including total samples, healthy outputs, failed sessions
- Provide detailed failure information with session IDs and error messages
- Generate re-run commands for samples with failed reviews
- Add percentage calculations for better statistical overview
- Include support for multiple output formats and detailed logging
* feat(benchmark): add LongMemEval output cleanup script and enhance golden check retry logic
- Added clean_sample_outputs.py script to remove generated LongMemEval files while preserving source inputs
- Implemented configurable retry mechanism in golden_check.py with exponential backoff strategy
- Added retry parameters (initial/max seconds and max attempts) to control failure recovery behavior
- Integrated asyncio support for asynchronous sleep during retry intervals
- Configured default retry settings in jinli_lme.yaml with 5s initial and 300s maximum intervals
- Preserved core files (query.json, answer.json, session/) while cleaning generated artifacts
* feat(benchmark): add AppleDouble file cleanup to sample output cleaner
- Remove AppleDouble files starting with '._' recursively including under session/
- Add is_under helper function to check if path is inside parent directory
- Track targets in set to avoid duplicate processing
- Include AppleDouble files in cleanup targets when not already covered by existing targets
- Maintain dry-run mode as default behavior with --apply flag for actual deletion
* refactor(benchmark): update LongMemEval sample output cleaning script
- Add time and Iterator imports for enhanced functionality
- Add --progress-every argument to control progress reporting frequency
- Replace is_under function with iter_sample_targets generator
- Implement detailed progress tracking with timing measurements
- Add sample-by-sample processing with elapsed time reporting
- Include AppleDouble file detection within session directory
- Update target counting and deletion statistics display
- Add conditional progress updates based on progress-every setting
- Improve dry-run mode with would-delete indication
* chore(benchmark): increase initial interval for session review step
- Changed START_INTERVAL_SECONDS from 1.0 to 3.0 seconds
- Adjusted timing parameters for better benchmark stability
* refactor(benchmark): implement coordinated retry mechanism for session reviews
- Add retry gate condition to coordinate concurrent review attempts
- Implement wait_for_healthy_start_slot to handle sequential retries
- Create mark_retrying and mark_recovered functions to track retry states
- Update reply_with_retry to accept index parameter for coordination
- Add has_prior_retry logic to prevent race conditions during recovery
- Ensure proper cleanup of retry state on success or failure
- Maintain backward compatibility while adding coordination features
* chore(benchmark): adjust session review start interval timeout
- Changed START_INTERVAL_SECONDS from 3.0 to 5.0 seconds
- Increased initial delay for session review benchmark step
- Updated timeout configuration for improved stability
* refactor(benchmark): update session review concurrency and throttling mechanism
- Replace global throttle with per-process concurrency control
- Add concurrency parameter with default value of 30 in config
- Add start_interval_seconds parameter with default value of 2 seconds
- Change default concurrency from 3 to 1 in command line interface
- Update documentation to reflect new throttling behavior
- Implement semaphore-based concurrency limiting for review tasks
- Modify retry mechanism to use local locking instead of global files
- Remove portalocker dependency for cross-process throttling
* refactor(config): update session review configuration and concurrency settings
- Removed deprecated retry configuration parameters from jinli_lme.yaml
- Increased MAX_CONCURRENCY from 30 to 60 in session_review.py
- Reduced START_INTERVAL_SECONDS from 2.0 to 1.0 in session_review.py
- Cleaned up redundant backend specifications in configuration file
- Simplified agent wrapper configurations by removing obsolete retry settings
* feat(benchmark): enhance LME auto memory step with advanced scheduling and error handling
- Add datetime parsing functionality for LongMemEval timestamps with regex pattern
- Implement configurable concurrency limits with MAX_CONCURRENCY of 60
- Introduce retry mechanism with exponential backoff for agent interactions
- Add session filtering based on date comparison with question_date validation
- Create rate limiting with start interval control between requests
- Implement sophisticated retry coordination using asyncio conditions
- Add comprehensive error tracking for failed and filtered session extracts
- Remove deprecated concurrency parameter from jinli_lme.yaml configuration
- Add structured output validation in session review step
- Include detailed metadata reporting with session statistics and errors
* fix(benchmark): adjust default concurrency for auto_memory job
- Changed default concurrency from 3 to 1 for auto_memory job to prevent API overload
- Updated help text to reflect new default value of 1 for concurrency parameter
- Modified documentation to clarify concurrency behavior varies by job type
* refactor(search): replace hardcoded candidate multiplier with constant
- Introduced _CANDIDATE_MULTIPLIER constant set to 10
- Replaced hardcoded factor of 5 with _CANDIDATE_MULTIPLIER in BM25 search
- Replaced hardcoded factor of 5 with _CANDIDATE_MULTIPLIER in vector search
- Updated test to verify both search steps use ten times limit for candidates
- Imported VectorSearchStep and Bm25SearchStep in test module
- Added comprehensive test case for candidate count calculation logic
* feat(lme): add data inspection error handling with fallback mechanism
- Implemented non-retryable data inspection error markers detection
- Added _is_data_inspection_error method to identify inspection failures
- Created fallback handling for data inspection errors in auto memory extraction
- Added fallback handling for data inspection errors in session review
- Extended failed extracts tracking with non-retryable and fallback flags
- Separated fallback extracts from regular failed extracts in reporting
- Enhanced error logging with specific data inspection failure messages
- Updated metrics to track fallback extractions and reviews separately
- Maintained existing retry logic for other exception types
* feat(benchmark): enhance session review statistics with fallback tracking
- Add support for identifying and listing non-retryable fallback reviews
- Introduce --list-fallback argument to display fallback review details
- Separate retryable failures from non-retryable fallbacks in reporting
- Track fallback samples and sessions separately from failed ones
- Update console output to show both retryable and non-retryable categories
- Include fallback details in JSON output with reasons and session info
- Modify failure counting logic to distinguish between retryable and fallback reviews
* feat(benchmark): add question_id tracking and enhanced fallback reporting
- Add question_id function to extract query.question_id from data
- Initialize question_id_by_id dictionary to store question IDs by index
- Store question_id for each sample during data processing
- Enhance fallback output to include question IDs and session information
- Format sample labels with question IDs when available
- Display session IDs associated with each fallback case
* feat(benchmark): add question_id support and improve bad sample reporting
- Add question_id_for function to extract question_id from multiple sources
- Add sample_label function to format samples as idx(question_id) when available
- Store question_id in data dictionary during processing
- Change bad_golden and bad_sessions to store full records instead of just indices
- Update list_bad output to show formatted labels with question_id information
- Improve error reporting with more detailed sample identification
* feat(benchmark): enhance golden check stats with structured output
- Add related_session_ids function to extract session IDs from verdict records
- Create grouped_records function to group records by question type
- Replace flat list output with JSON-formatted grouped records in list_bad option
- Replace flat list output with JSON-formatted grouped records in list_bad_sessions option
- Maintain Chinese labels while adding structured data presentation
- Improve readability of bad verdict record display with hierarchical grouping
* feat(benchmark): update data structure for question indexing
- Replace sample_label with _idx field for index tracking
- Add question_id field to store _question_id values
- Maintain backward compatibility with empty string defaults
- Preserve existing session_id functionality
- Update data mapping to include new fields in grouped results
* refactor(benchmark): streamline golden answer verification process
- Replace relevance filtering with comprehensive information extraction
- Remove is_relevant field and simplify session summary structure
- Change relevant_info to extracted_info for clarity
- Update golden check logic to work with full extractions instead of filtered summaries
- Simplify prompt instructions to focus on complete information extraction
- Remove redundant schema validation and structured output requirements
- Adjust statistics calculation to match new extraction approach
- Update metadata field names to reflect extraction rather than relevance checking
* feat(benchmark): add selective file deletion option to clean_sample_outputs
- Add --filename argument to delete only specific root-level files
- Modify iter_sample_targets function to accept optional filenames filter
- Implement validation for root-level filename constraints
- Update function calls to pass filenames parameter
- Add example usage for selective file deletion in documentation
* feat(benchmark): add error count metrics to golden check statistics
- Added golden_bad, session_bad, and both_bad calculation fields
- Updated console output format to include error counts per question type
- Modified table display to show both accuracy rates and error numbers
- Enhanced statistical summary with additional error breakdown metrics
* test(search): update search step tests with include_source parameter
- Added include_source=False parameter to VectorSearchStep initialization
- Added include_source=False parameter to Bm25SearchStep initialization
- Maintained existing RuntimeContext parameters for both search steps
- Updated test calls to match new constructor signature with include_source option
* feat(service): add CLI service for local job execution
- Introduce CliService to execute single jobs locally without serving ports
- Add prepare_start_config and should_precheck_start functions for CLI job setup
- Update reme start command to use CLI service when job argument is provided
- Change default service backend from http to cli in jinli_lme config
- Modify SearchStep to use constants and rename configuration parameters
- Add unit tests for CLI service functionality and configuration handling
- Update file extension support to include json format in addition to md and jsonl
* feat(search): add BM25 and vector search steps with configuration updates
- Add Bm25SearchStep and VectorSearchStep classes with tool context deduplication
- Register new search step components in index module
- Update configuration to use separate vector_search and bm25_search endpoints
- Modify LLM models from qwen3.7-plus/glm-5.1 to glm-5.2 variants
- Adjust search parameters and remove hybrid search implementation
- Configure embedding store as default in storage settings
- Remove auto-memory and file catalog configurations
- Update watch directories from multiple paths to session_dir only
* feat(agent): add tool result offloading and workspace management
- Add tool_results_dir configuration option for offloaded tool results storage
- Implement ToolResultOffloadMiddleware to persist large tool results to files
- Create WorkspaceBackend to standardize file operations across tools
- Add configurable builtin tools selection with sequential execution option
- Integrate middleware support for agent wrapper with offloading capability
- Update application initialization to create tool results directory
- Add safety mechanisms for filesystem operations with sanitized filenames
- Enhance agent wrapper with configurable working directory handling
- Upgrade agentscope dependency to version 2.0.4 for improved features
# Conflicts:
# reme/application.py
* feat(benchmark): add LongMemEval agentic search and result management
- Introduce AgenticAnswerStep for agent-based history search
- Add LmePrepareJudgeStep and LmeSaveResultStep for evaluation pipeline
- Implement AddDraftStep and ReadAllDraftStep for evidence accumulation
- Update configuration with new agent wrapper and search parameters
- Add comparison script for analyzing agent run differences
- Include documentation for LongMemEval failure analysis
- Enhance tool result offloading with skip options
- Modify search defaults and indexing behavior
* feat(agent): implement tool result offloading with system reminders
- Added tool_result_offload_message parameter to agent wrapper reply method
- Implemented configurable reminder template for offloaded tool results
- Created system reminder messages when tool results are offloaded to files
- Added Chinese user message template for agentic answer step
- Updated tool result offloading middleware to use custom reminder templates
- Enhanced agentic answer instructions to handle long tool results via draft storage
* feat(scripts): add LongMemEval results summarization tool
- Create summarize_lme_results.py script to analyze result JSON files
- Implement command line interface with answer id and dataset root options
- Add support for specifying index range with start and end parameters
- Include option to show failure details and non-successful completions
- Calculate completion statistics and accuracy metrics
- Display detailed breakdown of yes/no/other judgements
- Handle missing and unreadable result files gracefully
- Format output with percentages and comprehensive summary statistics
* feat(summarize_lme_results): add question type breakdown to result summary
- Import defaultdict from collections module
- Add by_type dictionary to track statistics by question type
- Count completed, yes, no, and other responses for each question type
- Display detailed breakdown table showing accuracy by question type
- Include question type column when processing judgements
- Print comprehensive summary with question type distribution
- Calculate and display accuracy percentage for each question type category
* feat(lme): switch to qwen3.7-max model and add shuffle functionality
- Changed default LLM model from glm-5.1 to qwen3.7-max in jinli_lme.yaml
- Added random module import for shuffle functionality
- Implemented --shuffle argument with BooleanOptionalAction for dataset shuffling
- Added --seed argument to control random seed for reproducible shuffling
- Applied random shuffle to dataset indices when shuffle is enabled
- Added console output showing shuffle operation and seed information
* fix(cli): set default random seed for shuffle functionality
- Changed default seed value from None to 42 for consistent shuffling behavior
- Ensures reproducible results when using shuffle option without explicit seed
- Maintains backward compatibility while providing deterministic defaults
* refactor(benchmark): update agentic answer guidelines for grounding
- Updated English instruction to emphasize strict grounding in retrieved context
- Modified Chinese instruction to stress evidence-based responses without inference
- Removed redundant conciseness requirement in both language versions
- Enhanced clarity on proper use of draft saving and retrieval mechanisms
- Strengthened emphasis against hallucination of unsupported facts
* refactor(benchmark): update agentic search instructions and configuration
- Replace separate vector_search and bm25_search with unified search tool
- Update agent instructions to use single search tool with multiple strategies
- Simplify Chinese instructions for search methodology
- Add comprehensive search tool configuration with hybrid vector/BM25 capabilities
- Increase model retry attempts from 1 to 3 for better reliability
- Remove redundant tool references from job_tools list
* feat(search): add configurable search limit with environment variable support
- Remove hardcoded limit and min_score parameters from config schema
- Increase LLM context size from 200000 to 1000000
- Add REME_SEARCH_LIMIT environment variable support for search configuration
- Implement command line argument --search-limit to override default search limit
- Add input validation to ensure search limit is positive
- Modify subprocess execution to pass environment variables
- Update search step to use dynamic default limit from environment or fallback to 5
* refactor(benchmark): remove agentic answer step and related configurations
- Removed AgenticAnswerStep class and its registration
- Deleted agentic_answer.yaml prompt configuration file
- Removed agentic answer related job definitions from jinli_lme.yaml
- Cleaned up tool result offloading middleware implementation
- Removed tool_results_dir configuration field from application config
- Deleted comparison and analysis scripts for agent runs
- Removed agentic answer step from LME init module exports
- Updated agent wrapper to remove tool result offloading functionality
- Removed unused imports and dependencies in agent wrapper module
* refactor(benchmark): remove unused LME result processing components
- Removed LmePrepareJudgeStep and LmeSaveResultStep classes from benchmark module
- Cleaned up imports and exports in lme module initialization
- Removed unused middleware configuration from agent wrapper
- Deleted obsolete result.py file containing deprecated result processing logic
- Simplified agent instantiation by removing middleware parameter
- Updated import statements to reflect removed dependencies
* refactor(index): remove unused search steps and update imports
- Remove Bm25SearchStep and VectorSearchStep from index steps module
- Remove unused prepare_start_config and should_precheck_start exports
- Move import statements to proper location in reme.py
- Update test module to use direct import path for CliService
- Remove vector_search and bm25_search configurations from jinli_lme.yaml
- Add workspace directory environment variable configuration
- Add docstring to getcwd method in agent wrapper
- Remove empty middleware list from agent wrapper initialization
* feat(index): add BM25 and vector search steps with tool context deduplication
- Add Bm25SearchStep for plain BM25 keyword search with tool_context deduplication
- Add VectorSearchStep for plain vector search with tool_context deduplication
- Implement tool context state management with TTL-based deduplication
- Add support for chunk deduplication across tool contexts within TTL window
- Update index steps module to include new search step classes
- Add test coverage for CLI metadata output functionality
- Refactor CLI service to remove unused show_status parameter
- Update documentation comments to reflect internal service configuration
* feat(steps): add Python code execution capability
- Introduce PythonExecuteStep to run Python code in subprocess
- Add configuration for python_execute step in jinli_lme.yaml
- Register python_execute in available tools list
- Implement timeout handling with default 60 second limit
- Capture stdout/stderr output and return code metadata
- Add comprehensive unit tests for execution scenarios
- Support workspace directory context for code execution
- Handle timeout errors and runtime exceptions gracefully
* refactor(python_execute): replace subprocess with asyncio for Python code execution
- Replace subprocess.run with asyncio.create_subprocess_exec for non-blocking execution
- Add _PythonResult dataclass to encapsulate execution results and timeout status
- Implement proper timeout handling with asyncio.wait_for and process.kill()
- Update metadata to include returncode and stderr when timeout occurs
- Convert synchronous _run_python method to asynchronous implementation
- Maintain backward compatibility while improving execution reliability
* refactor(python_execute): replace subprocess with asyncio for Python code execution
- Replace subprocess.run with asyncio.create_subprocess_exec for non-blocking execution
- Add _PythonResult dataclass to encapsulate execution results and timeout status
- Implement proper timeout handling with asyncio.wait_for and process.kill()
- Update metadata to include returncode and stderr when timeout occurs
- Convert synchronous _run_python method to asynchronous implementation
- Maintain backward compatibility while improving execution reliability
* fix(embedding): enforce strict dimension matching for embeddings
- Add _embedding_dim_matches method to validate embedding dimensions
- Reject embeddings with mismatched dimensions instead of padding/truncating
- Drop stale embeddings with wrong dimensions during loading and upsert operations
- Disable embedding store when query dimensions don't match configured dimensions
- Fail health checks when embedding dimensions don't match expected values
- Skip chunks with wrong dimensions during FAISS index rebuild
- Add comprehensive tests for dimension validation behavior
* refactor(file_store): simplify conditional checks in vector search and test assertions
- Combine multiple conditionals into single check for empty FAISS index
- Replace explicit empty list comparison with boolean check for node embedding calls
- Maintain same functional behavior while improving code readability
* fix(embedding): harden dimension validation helpers
* refactor(embedding): update embedding model initialization and session storage paths
- Remove unused inspect import from as_embedding module
- Pass dimensions directly to embedding model constructor instead of using parameters
- Update session state file paths to use mem_session directory instead of resource
- Add mem_session_dir configuration option to application config schema
- Update workspace directory creation to include new mem_session directory
- Change AgentScope and Claude Code session paths to use mem_session directory
- Move embedding dimensions from parameters to top-level configuration
- Update AgentScope dependency version from 2.0.3 to 2.0.4
- Update integration tests to reflect new session file location paths
* chore(version): bump version to 0.4.0.8
- Update __version__ from 0.4.0.7 to 0.4.0.8 in __init__.py
* refactor(search): replace time module with datetime for timestamp generation
- Removed unused time import
- Added static method _now_ts using datetime.timestamp
- Updated clock parameter to use _now_ts method instead of time.time
- Maintained same timestamp precision and functionality
* test(http): add tests for HTTP client display formatting
- Add test for default metadata hiding behavior in CLI output
- Add test for metadata display when show_metadata is enabled
- Verify _format_for_display method correctly formats response text
- Test both success case and metadata inclusion scenarios
* chore(build): remove longmemeval from gitignore
- Removed longmemeval directory from gitignore list
- Kept evaluation and datasets directories in ignore list
- Updated gitignore configuration for proper version control
* feat(search): add tool context deduplication and improve search configuration
- Modify _make_tool methods to accept and inject tool_context_id parameter
- Add tool_context_id handling in AS and CC agent wrappers
- Increase search candidate multiplier from 3.0 to 5.0 in default config
- Extend HTTP client timeout from 30s to 3600s
- Add tool context deduplication logic to prevent duplicate search results
- Implement TTL-based expiration for seen chunks in tool contexts
- Add comprehensive unit tests for tool context deduplication behavior
- Update .gitignore to exclude longmemeval directory
- Add time import for timestamp functionality in search step
* refactor(search): replace time module with datetime for timestamp generation
- Removed unused time import
- Added static method _now_ts using datetime.timestamp
- Updated clock parameter to use _now_ts method instead of time.time
- Maintained same timestamp precision and functionality
* feat: add start_date/end_date time filter support for search job
- Add _extract_date_from_path to extract validated YYYY-MM-DD from chunk paths
- Add start_date/end_date filtering in _matches_search_filter
- Implement progressive recall in FaissLocalFileStore.vector_search
- Promote start_date/end_date from context to search_filter in SearchStep
- Add start_date/end_date parameters to search job in default.yaml
- Add unit tests for date filter functionality
* fix: validate/normalize date filters and harden _extract_date_from_path
Address three code-review comments on the time_filter search feature:
1. Validate/normalize start_date and end_date before string comparison.
_matches_search_filter does lexicographic comparison against path_date
(always canonical YYYY-MM-DD). Raw caller values like '2026-2-28' or
'abc' would produce silently wrong results. Now SearchStep normalizes
valid dates via extract_daily_date (with strptime fallback for
non-zero-padded input) and silently ignores invalid dates with a
logger.warning, removing them from the filter.
2. Clarify behavior for paths without embedded dates.
Added optional strict_date_filter parameter (default False). When True
and at least one date bound is active, chunks whose path yields no date
(e.g. digest/personal/topic.md) are excluded. When False (default),
the existing behavior is preserved — dateless paths pass through.
3. Harden _extract_date_from_path against non-standard suffixes.
Previously parts[1].split('.')[0] accepted '2026-05-18.anything' as a
valid date. Now only exact 'YYYY-MM-DD' (dir) and 'YYYY-MM-DD.md'
(day-index) forms are accepted.
* fix(auto-memory): preserve message timestamps
* fix(auto-memory): infer daily date from messages
* feat(file_io): add strict date parsing and improve daily date handling
- Add new parse_daily_date function for strict YYYY-MM-DD validation
- Replace extract_daily_date with parse_daily_date for explicit date validation
- Change _messages_day to use max date instead of min for historical imports
- Reorder imports to maintain consistent module ordering
- Move session message saving after date validation in auto_memory
- Add comprehensive tests for invalid date rejection before saving
- Add tests for strict YYYY-MM-DD date format validation
- Update test names to reflect latest date behavior
---------
Co-authored-by: Ziyang Guo <121015044+RunMarshal@users.noreply.github.com>
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
* refactor(transfer): drop orphaned ingest step, make service discovery cross-platform
- Remove ingest step: superseded by auto_resource (drop files under
resource/ → watcher interprets them); its meta.json/<date>.md outputs
had no consumers and tripped the auto_resource watcher.
- Replace lsof/pgrep shell-outs in service_utils with psutil (per-process
enumeration, no root needed on macOS) for Windows/macOS/Linux support.
- Add cross-platform test coverage for _pid_on_port / _scan_reme_procs.
- Deps: +psutil, -filelock (only used by the removed ingest lock).
* chore(release): bump version to 0.4.0.5
* fix(mcp): resolve circular import issues and update dependencies
- Moved fastmcp imports inside functions to prevent circular dependencies
- Replaced _TRANSPORT_MAP with _VALID_TRANSPORTS set for transport validation
- Updated version number from 0.4.0.3 to 0.4.0.4
- Added claude-agent-sdk dependency to core optional dependencies
- Used TYPE_CHECKING imports for FastMCP related types
- Restructured transport mapping logic within function scope
- Fixed string annotation for CallToolResult type hints
* refactor(tests): update date handling in daily steps tests
- Replace _date.today() with timezone-aware now function
- Use Asia/Shanghai timezone for date formatting
- Change return format to use strftime instead of isoformat
- Import now function from reme.steps.evolve module
* refactor(tests): clean up unused imports in daily steps test
- Removed unused date import from datetime module
- Removed redundant pathlib Path import that was already imported later
- Kept necessary imports for asyncio, os, tempfile, warnings, and frontmatter modules
* test(daily_steps): update test to include application context for daily list step
- Add ApplicationContext initialization with temporary workspace directory
- Register file store component in application context
- Pass application context to DailyListStep constructor
- Maintain existing test assertion behavior for date metadata verification
* feat(file_io): add daily_write step for creating daily notes with conversation metadata
- Add DailyWriteStep class that delegates to write job for creating daily notes
- Register daily_write job in default configuration with proper parameters
- Include validation for name and session_id path components
- Add test coverage for daily_write functionality including metadata handling
- Preserve existing job execution method in application.py after repositioning
- Update base_step.py to use positional-only parameter syntax for job methods
- Import and expose DailyWriteStep in file_io module initialization
- Override reserved metadata keys (name, description, session_id, source_conversation) with fixed values
- Refresh daily index after successful write operation
- Generate proper source conversation links in markdown format
* feat(daily): refactor daily note system with enhanced metadata handling
- Introduce validate_filename_component function and export it
- Add _INDEX_HIDDEN_METADATA_KEYS to hide conversation metadata from index
- Update scan_notes to exclude hidden metadata keys from index rendering
- Modify auto_memory to use daily_write tool and manage session frontmatter
- Implement session note lookup and renaming based on frontmatter name
- Update daily_list to return flattened note metadata including session info
- Change daily_write to dispatch write step instead of running job
- Add test cases for updated daily note functionality and metadata handling
- Update version from 0.4.0.2 to 0.4.0.3
* fix(evolve): correct metadata update in auto memory response
- Fixed trailing comma issue in metadata dictionary update
- Ensured proper formatting of response metadata structure
- Maintained existing functionality while fixing syntax error
* refactor(auto_resource): replace daily_create with dynamic note management
- Remove DailyCreateStep and related exports from file_io module
- Replace static daily note creation with dynamic resource-linked card system
- Implement LLM-suggested naming with frontmatter-driven file management
- Add source_resource linking for tracking original files
- Introduce collision handling with hash-based suffixes
- Update documentation to reflect new resource card workflow
- Modify auto_resource prompts to use write/edit tools instead of daily_create
- Adjust test fixture comments to match new agent behavior
- Update framework diagrams and quick start examples accordingly
* feat(app): add version info to app initialization and update auto-memory logic
- Include version number in application startup logging
- Remove tool result truncation logic from auto-memory step
- Update auto-memory to exclude tool_result blocks from saved history
- Add test case to verify tool results are filtered out from message saving
- Update YAML prompts to clarify filename naming rules without dates
- Modify configuration to support new dispatch steps format with persistence control
* feat(auto_memory): add note modification tracking and optimize frontmatter updates
- Add _note_bytes and _note_modified methods to track actual file changes
- Optimize frontmatter updates by checking existing metadata before update
- Add modified flag to response metadata indicating actual note changes
- Update logging to include modified status in various operations
- Add comprehensive tests for modified/unmodified detection scenarios
- Enhance result hook logic to skip when no actual changes occur
- Refactor metadata handling to properly track creation vs modification status
* feat(evolve): enhance agent reply processing and logging capabilities
- Add agent_reply_result_text function to extract final user-visible text from agent replies
- Implement comprehensive logging throughout auto_memory, auto_resource, and dream modules
- Add max_units configuration option to limit extracted memory units
- Improve error handling and validation in auto_resource step
- Refactor dream extract step to respect max_units limit during processing
- Enhance summary rendering in dream finish step with detailed breakdown
- Add result hook functionality for embedding hosts integration
- Implement loose resource filename handling for root-level resources
- Update test cases to reflect new functionality and improved error messages
* test(background-steps): update fake upsert function to include created parameter
- Modified fake_upsert function to accept 'created' parameter instead of '_created'
- Added 'created' field to captured dictionary in fake_upsert function
- Included 'created': True in the expected response dictionary for test case
- Updated test assertion to match new parameter structure
- Implement _backfill_missing_embeddings method to handle chunks without embeddings
- Add logic to identify and process chunks that predate embedding feature
- Integrate backfill process into store loading sequence
- Add proper error handling and logging for backfill operations
- Create unit test for embedding backfill functionality
- Ensure backfilled embeddings are properly persisted to storage
* chore(logging): change info logs to debug level for data loading operations
- Changed stopwords loading log from info to debug level
- Changed file catalog nodes loading log from info to debug level
- Changed file graph nodes loading log from info to debug level
* feat(dream): add dream schema definitions and enum for auto-dream functionality
- Add DreamBucketEnum with procedure, personal, and wiki values
- Create comprehensive dream-related Pydantic models including DreamUnit,
DreamTopic, DreamExtractOutput, IntegrateOutcome, TopicSelectionOutput,
ProactiveResult, and DreamState
- Move schema definitions from local step module to shared schema package
- Update dream extraction and integration steps to use new enum-based
bucket validation
- Initialize digest directories for each dream bucket type
- Enhance embedding store health check with workspace directory logging
* refactor(tests): update DreamState import path in test_auto_dream.py
- Move DreamState import from reme.steps.evolve.dream.schema to reme.schema
- Maintain same functionality with updated module reference
- Align import with new schema location in project structure
* fix(core): update version number to 0.4.0.1
- Incremented version from 0.4.0.0 to 0.4.0.1 in __init__.py
* fix(index): remove stopwords path from tokenizer config and add keyword index repair
- Remove stopwords_path from tokenizer config to prevent index forking by install path
- Add _sync_keyword_index_from_chunks method to repair keyword index when persisted state mismatches
- Implement test for keyword index repair from persisted chunks when missing
- Add test to verify tokenizer fingerprint ignores stopwords absolute path
- Update version from 0.4.0.1 to 0.4.0.2
* feat(dream): add scan_days parameter to dream extraction process
- Add scan_days configuration option to default.yaml with default value of 2
- Implement recent_dates utility function to calculate date ranges for scanning
- Modify DreamExtractStep to scan multiple days based on scan_days parameter
- Update dream extraction to process files across multiple dates instead of single day
- Extend DreamState schema to include dates and scan_days fields
- Update DreamTopicsStep to handle multi-day topic processing
- Modify finish step to checkpoint files from all scanned dates
- Add comprehensive tests for multi-day scanning functionality
- Update prompt templates to include scan dates information
- Refactor topics writing logic to target specific date rather than current date
* docs: rename vault_dir to workspace_dir in documentation and examples
* refactor(extract): format long method call across multiple lines
* refactor(extract): format system prompt parameters for better readability
* refactor(steps): update naming conventions in components and configuration
Updated naming conventions across multiple files, changing colon-separated names to underscore-separated format, and added new step definitions along with documentation updates.
Key changes:
- Replaced `Synchronizer` with `AutoMemory` as the counterpart component for cold-write operations
- Updated naming conventions in all related configuration files (e.g., `frontmatter:read` → `frontmatter_read`)
- Added new step definitions such as `submit_slug_updates` and `auto_memory`
- Updated relevant documentation
- Modified log output format for improved readability
* refactor(evolve): Refactor the auto-memory module and update related configurations
- Remove the old slug update commit step file
- Add new auto-memory planner and writer steps
- Update __init__.py to export the new step classes
- Modify the auto_memory configuration structure in default.yaml
- Update the slug field description for clearer explanation of its purpose
* up
* up
* refactor(tests): Move unit test directory from `tests4/unittest` to `tests4/unit`
Additionally, the assertion logic in test files has been updated: direct comparisons of `payload["notes"]` have been replaced with checks verifying the presence of paths and metadata within the response content. Furthermore, some test expectations have been simplified—for example, using `count` instead of asserting against specific note lists.
Specific changes include:
- Updating workflow configurations to align with the new test directory structure
- Modifying assertions across multiple test methods to make them more flexible and maintainable
- Cleaning up and optimizing parts of the test code structure
This is a comprehensive test refactoring effort aimed at improving test readability and robustness.
* Refactor(steps): Update memory writing logic and optimize JSON schema structure
Improved the write strategy description in `auto_memory_writer.yaml` to emphasize using `edit` over `write`.
Adjusted the `json_schema` structure in `base_step.py` to support the new function definition format.
Also corrected grammatical issues in the related documentation.
* Fix: Improve frontend data parsing error handling and update test files
Added capture and handling logic for YAML parsing exceptions, providing more detailed error messages when frontend data format issues occur. Also corrected the description text in a test file.
* feat(seekdb): add Seekdb file and vector stores with pyseekdb>=1.2.0
* refactor(seekdb): add pyseekdb_conn and remote-only host/port config
* refactor(embedding): remove env fallbacks from BaseEmbeddingModel; pass credentials in tests
* refactor(seekdb): drop tenant from client kwargs; default database test and empty password
* fix(deps): gate pyseekdb to Python >=3.11 for CI 3.10 compatibility
* fix(seekdb): satisfy pre-commit pylint and formatting for seekdb stores
* fix(reme_light): dedupe default watch paths on case-insensitive filesystems
On Windows NTFS and macOS HFS+, ``MEMORY.md`` and ``memory.md`` resolve to
the same physical file. ``ReMeLight.__init__`` hardcoded both spellings in
the default ``watch_paths`` list, so the memory markdown file was indexed
twice on those filesystems, wasting embedding calls and producing duplicate
search hits.
Dedupe the default candidate list using ``os.path.normcase`` as the
comparison key. On case-sensitive filesystems normcase is the identity
function, so both spellings continue to be watched there. The original
path strings are preserved, the caller-supplied ``watch_paths`` path is
untouched, and only the built-in fallback is affected.
Fixes#228
* refactor(reme_light): simplify watch path dedup via existence check
Replace the os.path.normcase-based dedup loop with a direct exists()
check that picks one of MEMORY.md / memory.md. On case-insensitive
filesystems both spellings resolve to the same file so exists() returns
true for both, naturally avoiding a duplicate watch — including on
macOS where os.path.normcase is the identity function and the previous
approach silently did nothing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* feat(vector_store): add OceanBase as a VectorStore
* refactor(obvec): make it cleaner
* docs: add obvec related info
* refactor: minor update
* refactor: clean code and pass lint
* docs: remove unrelated edit
* docs: minor update
* refactor(file_store): simplify ChromaDB client initialization and improve file truncation logic
- Remove shutil import and _create_chroma_client method from chroma_file_store.py
- Directly initialize ChromaDB PersistentClient in start method without retry logic
- Reduce DEFAULT_MAX_BYTES from 100KB to 50KB in file_utils.py
- Update truncation notice format to provide clearer continuation instructions
- Add _truncate_fresh and _retruncate functions for better text truncation handling
- Replace inline truncation logic with dedicated function calls in file_utils.py
- Rename skills_tool_ids to md_file_tool_ids in tool_result_compactor.py
- Update file detection logic to identify any .md files instead of only skill.md
- Create comprehensive unit tests for truncation functionality in test_truncate_text_output.py
* chore(version): bump version to 0.3.1.6
- Update __version__ from 0.3.1.5 to 0.3.1.6 in __init__.py
* refactor(core): replace text truncation utilities with new marker system
- Remove old truncate_text_utils module and its exports
- Replace TRUNCATION_MARKER_START with _TRUNCATION_NOTICE_MARKER constant
- Update as_msg_stat.py to split content using new marker format
- Modify FileIO tool to use TRUNCATION_NOTICE_MARKER for continuation hints
- Change is_truncated function checks to use marker presence detection
- Move transformers dependency from main deps to light extra dependencies
- Update tool result compactor tests to verify marker instead of is_truncated calls
* feat(file_io): enhance file operations with path resolution and append functionality
- Add expanduser() to resolve file paths with ~ symbol
- Implement proper file existence and type validation in update_file
- Add new append_file method to append content to files
- Update truncation notice format for better readability
- Fix typo in error message from "provide" to "provided"
- Update transformers dependency in pyproject.toml
- Remove duplicate transformers dependency from light extras
* refactor(file_io): disable pylint too-many-return-statements warning
* perf(file_watcher): increase default polling delay and optimize watcher configuration
- Increased default poll_delay_ms from 1000ms to 2000ms to reduce CPU usage
- Removed force_polling parameter as it's no longer needed with updated polling strategy
- Simplified async watch configuration by removing conditional force_polling logic
- Reduced overall system resource consumption during file watching operations
* refactor(memory): update conversation log documentation in memory summary
- Changed "Raw conversation logs" to "Earlier conversation logs" for clarity
- Added warning note about potentially large dialog file sizes
- Improved formatting with additional line break for better readability
- Maintained existing compressed summary integration unchanged
* feat(memory): add long-term memory support to file-based memory system
- Initialize _long_term_memory attribute as empty string
- Add memories section to content when long-term memory exists
- Consolidate summary and memories into single user message
- Format memories with markdown header # Memories
- Maintain existing compressed summary functionality
- Join multiple content parts with double newlines
* style(memory): update message formatting and improve logging
- Change default include_thinking parameter to True in as_msg_handler.py
- Replace angle brackets with square brackets for block formatting in as_msg_stat.py
- Add newline replacement in text truncation method in as_msg_stat.py
- Add loading duration timing to embedding cache loading in base_embedding_model.py
- Replace XML-style tags with markdown headers in compactor.py conversation format
- Update compactor.yaml prompts to reference markdown-style headers instead of XML tags
- Modify summarizer.py to use markdown-style conversation header format
* refactor(file-watcher): replace scan_on_start with rebuild_index_on_start parameter
- Replace scan_on_start and clear_on_start boolean parameters with single rebuild_index_on_start
- Update BaseFileWatcher constructor to use rebuild_index_on_start instead of two separate flags
- Modify initialization logic to clear and rescan when rebuild_index_on_start is True
- Remove scan_on_start parameter from CLI and light configuration files
- Update documentation to remove scan_on_start from quick start guides
- Rename all test methods and classes from scan_on_start to rebuild_index_on_start
- Add timezone-aware datetime helper method to summarizer component
- Format log message with proper line breaks for readability
* fix(core): resolve file watcher initialization issue and update version
- Fixed file watcher task creation to properly handle rebuild index on start logic
- Moved initialization and watch loop into async function to ensure proper execution order
- Updated package version from 0.3.1.1 to 0.3.1.2
- Added missing comma in embedding model logging statement
* fix(core): reduce max formatter text length limit
- Changed _DEFAULT_MAX_FORMATTER_TEXT_LENGTH from 2000 to 1000
- Updated constant value in as_msg_stat.py schema module
* fix(file-watcher): change default rebuild index behavior on start
- Changed rebuild_index_on_start parameter default from False to True
- This ensures index is rebuilt by default when file watcher starts
- Maintains consistent state initialization for file watching operations
* feat(compactor): add return_dict option and improve summary validation
- Add _is_valid_summary function to validate summary content format
- Introduce return_dict parameter to return structured results with validation
- Update prompt templates with clearer task descriptions and formatting rules
- Refactor update_user_message prompts to combine prefix and suffix logic
- Return dictionary with user_message, history_compact, and is_valid fields when enabled
- Add proper error handling for exception cases in memory compaction
- Maintain backward compatibility with string return when return_dict=False
* feat(memory): add thinking block configuration option
- Add add_thinking_block parameter to compactor component
- Pass include_thinking flag to message formatting in compactor
- Add add_thinking_block parameter to reme_light compact function
- Add add_thinking_block parameter to reme_light summarize function
- Add add_thinking_block parameter to summarizer component
- Pass include_thinking flag to message formatting in summarizer
- Remove previous-summary tags from compressed summary format
