From 26cb5ca62fe571d499f10ac746c5563663169a3d Mon Sep 17 00:00:00 2001
From: jinliyl <6469360+jinliyl@users.noreply.github.com>
Date: Fri, 19 Jun 2026 01:47:14 +0800
Subject: [PATCH] Dev/0618 (#282)

* feat(core): enhance agent wrapper with session management and stream chunks

- Add AddStep for simple arithmetic operations
- Implement session persistence with AsStateHandler for AgentScope
- Introduce stream chunk conversion for unified event handling
- Add file session store for Claude Code agent backend
- Support skill integration and permission handling in agent wrappers
- Add proactive and daily topics features to auto dream step
- Refactor auto memory and auto resource steps to use job tools
- Update application shutdown sequence for proper resource cleanup
- Enhance environment loading utilities with parse_env_file function
- Add comprehensive error handling and validation for session IDs

* refactor(embedding): simplify model initialization and update component management

- Replace individual _start methods with shared model_cls attribute pattern
- Remove redundant _close methods from embedding model wrappers
- Add application-level component update capability via update_component method
- Simplify BaseAsLLM _start method with early return when model exists
- Remove unused skill instruction template from agent wrapper
- Restructure CronJob to execute its own steps instead of dispatching external jobs
- Update cron job configuration format to use steps instead of dispatch targets
- Refactor cron job tests to match new execution model
- Remove deprecated dispatch_step/dispatch_job functionality from cron job

* feat(agent): add session management and cleanup functionality

- Added session_dir configuration field for persisted agent sessions
- Included session directory in vault initialization process
- Implemented session retention period with configurable days
- Added automatic cleanup of expired session files
- Introduced session cleanup flag to prevent duplicate operations
- Updated project path calculation to use vault path directly
- Removed fallback streaming implementation from base class

* feat(agent): add session management and cleanup functionality

- Added session_dir configuration field for persisted agent sessions
- Included session directory in vault initialization process
- Implemented session retention period with configurable days
- Added automatic cleanup of expired session files
- Introduced session cleanup flag to prevent duplicate operations
- Updated project path calculation to use vault path directly
- Removed fallback streaming implementation from base class

* refactor(application): restructure job startup order and improve error handling

- Change job startup sequence from background-last to base-stream-background-cron
- Import specific job types (BackgroundJob, CronJob, StreamJob) instead of generic BaseJob
- Update isinstance checks for proper job type identification
- Implement robust error handling during component closure with preserved exceptions
- Modify job merging to combine config and call-time kwargs in BaseJob and StreamJob
- Update service job registration to return boolean success indicators
- Add timezone support for cron job scheduling
- Enhance keyword index persistence with component-specific filenames
- Add comprehensive file store consistency tests and search filtering capabilities
- Include tokenizer stopwords in package distribution
- Fix prompt handler validation behavior and error messages

* fix(core): handle exceptions during application startup and improve validation

- Add exception handling around component startup to close started components on failure
- Replace assertions with runtime checks in claim_channel step for Python -O compatibility
- Add input validation for config parser including empty keys and non-mapping roots
- Enhance environment variable expansion to convert scalar types
- Add support for relative config file paths by searching in config directory
- Validate 2D array requirements in batch cosine similarity function
- Update channel notify step to return proper response objects
- Pass client-specific arguments through CLI to HTTP client initialization
- Add comprehensive tests for error conditions and edge cases

* feat(graph): add Neo4j backend support with enhanced health monitoring

- Implement Neo4jFileGraph component with connection constraints and async operations
- Add cached count tracking for nodes, edges, and virtual nodes in Neo4j backend
- Update health check to include Neo4j status reporting with memory usage
- Modify LLM demo steps to always register add tool without conditional flag
- Enhance AddStep to handle numeric string conversion and input validation
- Add comprehensive unit tests for Neo4j integration and error handling scenarios
- Remove deprecated use_add_tool parameter from LLM demo components
- Update integration tests to reflect simplified tool registration approach

* feat(file_io): enhance file I/O operations with path validation and large file handling

- Add resolve_path function with comprehensive path validation and security checks
- Implement read_file_lines_safe for efficient reading of large files by line ranges
- Integrate path validation across all file I/O operations to prevent directory traversal
- Add proper error handling for invalid paths and file access issues
- Enhance daily index operations with path resolution and error reporting
- Add 'changed' field to index responses to track file modification status
- Update file listing operations to use secure path resolution
- Add support for JSONL files in default scanning operations
- Improve move and delete operations with proper path validation
- Add comprehensive path validation tests and security checks

* refactor(steps): update file I/O and prompt handling implementations

- Add module docstring to file_io/__init__.py
- Remove unused validate parameter from prompt_format method
- Update import path from reme.reme to reme4.reme in common_utils.py
- Add missing docstrings to test classes and methods
- Remove deprecated test_format_missing_variable_no_validate test
- Simplify assertion in test_job.py using not operator
- Update import statement in test_utils.py for common_utils
- Add docstrings to dummy classes and functions in tests
- Rename variable in get_node_embeddings for clarity

* refactor(steps): restructure step modules and update change handling

- Split monolithic steps module into channel, common, evolve, file_io, index, and transfer submodules
- Replace ScanStoreChangesStep and ScanCatalogChangesStep with unified InitChangesStep
- Remove ForeachDispatchStep and replace with direct dispatch_steps mechanism in InitChangesStep
- Update configuration to use new init_changes_step with dispatch_steps pattern
- Add coalesce_changes utility for collapsing duplicate file change events
- Enhance AutoResourceStep to handle batch changes instead of single file operations
- Introduce ClearStoreStep to replace ClearAndScanStep functionality
- Add async locks to LocalFileCatalog for thread-safe operations
- Update documentation to reflect new directory structure and session organization

* test(steps): add comprehensive unit tests for background steps and search functionality

- Add new test_background_steps.py with initialization and dispatch update tests
- Add test_index_update_loop_init_dispatch_updates_store_across_batches function
- Add test_digest_watch_loop_init_dispatch_updates_named_catalog_and_logs function
- Create new test_search_step.py with complete SearchStep unit test coverage
- Implement FakeSearchStore for isolated SearchStep testing without external dependencies
- Add hybrid search RRF merging test with vector and keyword result fusion
- Include keyword-only search test with min_score filtering functionality
- Add empty query validation test with early failure mechanism
- Test vector and keyword search method calls with proper parameter passing
- Verify score handling and result ranking in hybrid search scenarios

* refactor(file_io): remove session_agent prefix from daily note filenames

- Removed 'session_agent_' prefix from daily note file naming pattern
- Updated all references in auto_dream, auto_memory, auto_resource, and daily_steps
- Modified config documentation to reflect new filename pattern
- Changed session file storage location in auto_memory to reme_session/dialog/
- Added validate_session_id and write_file_safe imports to file_io module
- Updated tests to match new filename convention without 'session_agent_' prefix
- Fixed day index refresh logic to properly update note count descriptions
- Adjusted proactive step to use new file path pattern for session notes

* feat(auto_resource): change resource processing to use same-name daily notes

- Replace MD5-based session ID generation with UUID5 for agent sessions
- Compute note stem from resource filename instead of hashing for daily note naming
- Update delete handler to use note stem instead of session ID for file lookup
- Modify upsert handler to use note stem as session ID parameter
- Change execute method to require changes as list of dictionaries
- Update test cases to use changes array instead of individual file_path and change parameters
- Adjust test assertions to verify same-name daily note creation and modification
- Refactor session state storage to use AgentScope format and location
- Remove deprecated session_state file handling in favor of new note system

* docs(structure): update resource naming convention in documentation

- Change resource naming from hash-based to stem-based format
- Update file path references from resource_{hash(resource_name)}.md to {resource_stem}.md
- Modify documentation to reflect new resource storage structure
- Adjust auto-resource saving location to use resource stem instead of hashed name

* refactor(evolve): split auto_dream into multi-step pipeline with dedicated dream modules

- Replace single AutoDreamStep with 4-step pipeline: extract, integrate, topics, finish
- Create new dream module structure under reme4/steps/evolve/dream/
- Remove old dream.py, auto_dream.py, and daily_topics.py files
- Add DreamExtractStep, DreamFinishStep, DreamIntegrateStep, DreamTopicsStep, ProactiveStep
- Update evolve init to import new dream step classes instead of old modules
- Document complete auto_dream logic breakdown and refactoring plan in markdown
- Consolidate dream-related functionality into focused, testable components
- Maintain LLM integration while improving step separation and error handling

* refactor(dream): update system prompts and integration logic

- Replace 'Phase 1/2' terminology with descriptive agent names in prompts
- Update extraction and integration prompts to clarify unit processing flow
- Add explicit instructions about provenance tracking and wikilink handling
- Enhance validation for source material references in personal preferences
- Add fallback mechanism to preserve topics when selection fails
- Include punctuation handling guidelines for YAML parsing
- Add comprehensive tests for wikilink graph relationships
- Create new configuration file with complete service definitions

* refactor(steps): remove auto_dream and dream steps, update config access

- Removed auto_dream step implementation and its associated logic
- Removed dream step implementation including extract and integrate phases
- Replaced direct app_context.app_config access with config_value method
- Added config_value helper method to BaseStep for unified config access
- Updated auto_memory to use session_dir config and add source conversation links
- Modified auto_resource to use config_value for directory paths
- Updated daily_* steps to use config_value for daily directory
- Added exception raising in bm25_index save method on failure
- Introduced SOURCE_CONVERSATION_KEY constant for tracking source sessions

* refactor(embedding): update embedding component to use credential-based initialization

- Replace model_cls with credential_cls for authentication handling
- Update configuration structure to separate credential and parameters
- Modify health check to access model name through new attribute path
- Change input parameter from 'text' to 'inputs' for generic handling
- Add credential initialization and parameter parsing in _start method
- Update agentscope dependency to version 2.0.2
- Remove direct model class references in favor of credential-based lookup