mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-08-28 05:25:04 +00:00
900 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Sen Huang
|
8042f74b6f
|
docs: add comprehensive documentation for auto-dream, auto-link, and auto-resource flows (#343) | ||
|
jinliyl
|
b5e0ec2d8d
|
Modify budget calculation for text limit safety margin
Adjust budget calculation to use 92% margin for token estimation. |
||
|
jinliyl
|
07d4527a0d
|
docs(agents): update coding conventions for state persistence (#342)
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
- Add guideline that steps should be stateless - Specify storing persistent state in self.app_context.metadata - Clarify avoiding state storage on step instances |
||
|
jinliyl
|
bf7ca17705
|
feat(benchmark): add LongMemEval golden answer validation (#335)
* feat(benchmark): add golden answer validation and session review for LongMemEval - Introduce GoldenCheckStep to validate LongMemEval golden answers using structured verdicts - Add SessionReviewStep to extract query/answer-relevant evidence from all sessions - Implement concurrent session processing with configurable concurrency limits - Create check_golden job configuration with lme_review and lme_judge agent wrappers - Add Qwen3.7-plus model configuration for enhanced processing capabilities - Include python_execute tool integration for agent-based reasoning and date validation - Generate comprehensive JSON output with session summaries and validation verdicts - Add run_check_golden.py script for batch processing across all LongMemEval samples - Configure proper logging initialization with console and file output options - Update component registry and file I/O modules to support new benchmark features * feat(scripts): add script to summarize LongMemEval check_golden verdicts - Parse check_golden.json files across all LongMemEval samples - Calculate accuracy metrics for golden answers and session IDs - Provide breakdown by question type with percentage calculations - Add command line options for listing bad samples and JSON output - Include progress tracking showing completed vs pending samples - Display confidence scores and date sanity checks statistics * refactor(benchmark): move golden check scripts to longmemeval directory - Moved run_check_golden.py from scripts/ to benchmark/longmemeval/ - Moved stats_check_golden.py from scripts/ to benchmark/longmemeval/ - Updated path resolution to use parents[2] instead of parent.parent - Added new --list-run-failed option to stats script - Added logging directory constant and functions for tracking launched samples - Enhanced stats output with launched count and run failure information - Improved error reporting with run failure details and log file paths * feat(benchmark): add LongMemEval agentic answer workflow with session extraction - Add LmeAgenticAnswerStep, LmeAutoMemoryStep, and LmeExtractSessionStep to __init__.py - Create shared helper render_with_source for displaying search results with session_id - Implement agentic_answer step with vector_search, bm25_search, and extract_session_by_id tools - Add auto_memory step to convert each session into search-friendly daily notes - Create extract_session step to retrieve and analyze raw session content by session_id - Update jinli_lme.yaml with auto_memory, vector_search, bm25_search, and agentic_answer jobs - Configure lme_memory, lme_extract, and lme_agentic_answer agent wrappers - Enhance search steps with include_source option to show session_id metadata - Add proper session_id tracking and collision handling in daily note generation * feat(benchmark): add LongMemEval agentic answer evaluation pipeline - Add session_id tracking to agentic_answer.py result metadata - Introduce run_agentic_answer.py driver for complete pipeline execution - Implement auto_memory, update_index, and agentic_answer job orchestration - Add concurrent execution with configurable limits and staggering - Create aggregation script for collecting tool-call trails and results - Add stats_agentic_answer.py for comprehensive result analysis - Implement resume capability with existing output detection - Generate aggregate.json with per-sample breakdown and tool call summaries * feat(steps): add ClearPathsStep for cleaning workspace outputs before rebuild - Introduce ClearPathsStep to remove stale workspace files/directories - Add support for specifying paths and config_keys as targets to clear - Implement safety checks to prevent deletion of files outside workspace - Add logging for cleared paths and warnings for invalid paths - Configure clear_paths_step in jinli_lme.yaml to clean daily_dir - Add clear_paths_step to clean mem_answer.json before rebuilds * feat(benchmark): add resume functionality to agentic answer runner - Replace --force flag with --resume flag for controlling job execution - By default every job reruns with clean rebuild behavior using config clear steps - Add --resume option to skip samples whose output already exists and continue interrupted batches - Update documentation to reflect new default clean rebuild behavior - Modify job skipping logic to honor resume flag instead of force flag - Update dry-run output to show correct todo jobs based on resume status - Change default example command to use --resume for continuing interrupted runs * feat(benchmark): generate JSONL output for check golden records - Add write_check_golden_list function to create JSONL file - Write all readable check_golden records as JSONL format - Include check_golden_list path in stats output - Display generated JSONL file path in summary report - Maintain UTF-8 encoding with non-ASCII character support * refactor(benchmark): rename answer judge step and integrate LME LLM judge - Rename AnswerJudgeStep to LmeLlmJudgeStep and update imports - Add new llm_judge configuration in jinli_lme.yaml - Update run_agentic_answer.py to include llm_judge in pipeline - Modify LmeLlmJudgeStep to read from query.json and answer.json - Write LLM judgement results back to mem_answer.json - Add command line options for start/end sample range selection - Update aggregate.json generation to include LLM judgement data - Add resume capability for llm_judge job based on judgement presence * refactor(benchmark): rename answer judge step and integrate LME LLM judge - Rename AnswerJudgeStep to LmeLlmJudgeStep and update imports - Add new llm_judge configuration in jinli_lme.yaml - Update run_agentic_answer.py to include llm_judge in pipeline - Modify LmeLlmJudgeStep to read from query.json and answer.json - Write LLM judgement results back to mem_answer.json - Add command line options for start/end sample range selection - Update aggregate.json generation to include LLM judgement data - Add resume capability for llm_judge job based on judgement presence * feat(steps): add wait_for_paths_step to block until workspace files exist - Introduce WaitForPathsStep class that polls for required workspace-relative paths - Add step registration with 'wait_for_paths_step' backend identifier - Implement path validation to ensure targets are within workspace boundaries - Add polling mechanism with configurable intervals via poll_seconds parameter - Include logging functionality with log_every_seconds parameter for status updates - Add metadata tracking of waited paths and duration in response object - Register step in index module and expose in public API - Configure step in jinli_lme.yaml to wait for session_review.json before golden check - Add script rename from run_check_golden.py to run_golden_check.py with enhanced options * feat(benchmark): enhance longmemeval benchmarking with concurrency and progress tracking - Add benchmark extra dependency group with portalocker requirement - Introduce concurrent execution support for golden_check and session_review workflows - Add progress reporting interval option with real-time status updates - Implement global throttling mechanism for session review requests using file locks - Enhance golden check validation with current schema verification - Add active task tracking and graceful shutdown handling - Rename check_golden scripts to golden_check for consistency - Update statistics reporting with correct/incorrect terminology instead of reasonable - Add stale format detection and compatibility handling for verdict fields - Include both_correct rate calculation in accuracy metrics - Add concurrency and staggering options for better resource management * ci(workflow): add Windows smoke test workflow - Create new workflow file .github/workflows/windows-smoke.yml - Configure workflow to trigger on push and pull request events - Set up Python environment with version 3.11 - Install package dependencies using pip - Run version job as smoke test for CLI functionality - Enable concurrency control to prevent duplicate runs - Use matrix strategy for Python version testing * feat(benchmark): add retry mechanism and health check for session review - Added retry configuration options (retry_initial_seconds, retry_max_seconds, retry_max_attempts) to jinli_lme.yaml - Implemented exponential backoff retry logic with configurable parameters in session_review step - Added output_is_healthy function to verify session_review.json integrity and absence of failed reviews - Updated resume functionality to skip only healthy outputs instead of all existing files - Integrated JSON parsing and validation to check for failed reviews in output files - Enhanced error handling and logging for retry attempts and recovery scenarios * feat(benchmark): add LongMemEval session review statistics script - Create stats_session_review.py to summarize session_review.json artifacts - Add command line options for listing failed, missing, and run failed samples - Implement JSON output mode for programmatic consumption - Calculate and display health statistics including total samples, healthy outputs, failed sessions - Provide detailed failure information with session IDs and error messages - Generate re-run commands for samples with failed reviews - Add percentage calculations for better statistical overview - Include support for multiple output formats and detailed logging * feat(benchmark): add LongMemEval output cleanup script and enhance golden check retry logic - Added clean_sample_outputs.py script to remove generated LongMemEval files while preserving source inputs - Implemented configurable retry mechanism in golden_check.py with exponential backoff strategy - Added retry parameters (initial/max seconds and max attempts) to control failure recovery behavior - Integrated asyncio support for asynchronous sleep during retry intervals - Configured default retry settings in jinli_lme.yaml with 5s initial and 300s maximum intervals - Preserved core files (query.json, answer.json, session/) while cleaning generated artifacts * feat(benchmark): add AppleDouble file cleanup to sample output cleaner - Remove AppleDouble files starting with '._' recursively including under session/ - Add is_under helper function to check if path is inside parent directory - Track targets in set to avoid duplicate processing - Include AppleDouble files in cleanup targets when not already covered by existing targets - Maintain dry-run mode as default behavior with --apply flag for actual deletion * refactor(benchmark): update LongMemEval sample output cleaning script - Add time and Iterator imports for enhanced functionality - Add --progress-every argument to control progress reporting frequency - Replace is_under function with iter_sample_targets generator - Implement detailed progress tracking with timing measurements - Add sample-by-sample processing with elapsed time reporting - Include AppleDouble file detection within session directory - Update target counting and deletion statistics display - Add conditional progress updates based on progress-every setting - Improve dry-run mode with would-delete indication * chore(benchmark): increase initial interval for session review step - Changed START_INTERVAL_SECONDS from 1.0 to 3.0 seconds - Adjusted timing parameters for better benchmark stability * refactor(benchmark): implement coordinated retry mechanism for session reviews - Add retry gate condition to coordinate concurrent review attempts - Implement wait_for_healthy_start_slot to handle sequential retries - Create mark_retrying and mark_recovered functions to track retry states - Update reply_with_retry to accept index parameter for coordination - Add has_prior_retry logic to prevent race conditions during recovery - Ensure proper cleanup of retry state on success or failure - Maintain backward compatibility while adding coordination features * chore(benchmark): adjust session review start interval timeout - Changed START_INTERVAL_SECONDS from 3.0 to 5.0 seconds - Increased initial delay for session review benchmark step - Updated timeout configuration for improved stability * refactor(benchmark): update session review concurrency and throttling mechanism - Replace global throttle with per-process concurrency control - Add concurrency parameter with default value of 30 in config - Add start_interval_seconds parameter with default value of 2 seconds - Change default concurrency from 3 to 1 in command line interface - Update documentation to reflect new throttling behavior - Implement semaphore-based concurrency limiting for review tasks - Modify retry mechanism to use local locking instead of global files - Remove portalocker dependency for cross-process throttling * refactor(config): update session review configuration and concurrency settings - Removed deprecated retry configuration parameters from jinli_lme.yaml - Increased MAX_CONCURRENCY from 30 to 60 in session_review.py - Reduced START_INTERVAL_SECONDS from 2.0 to 1.0 in session_review.py - Cleaned up redundant backend specifications in configuration file - Simplified agent wrapper configurations by removing obsolete retry settings * feat(benchmark): enhance LME auto memory step with advanced scheduling and error handling - Add datetime parsing functionality for LongMemEval timestamps with regex pattern - Implement configurable concurrency limits with MAX_CONCURRENCY of 60 - Introduce retry mechanism with exponential backoff for agent interactions - Add session filtering based on date comparison with question_date validation - Create rate limiting with start interval control between requests - Implement sophisticated retry coordination using asyncio conditions - Add comprehensive error tracking for failed and filtered session extracts - Remove deprecated concurrency parameter from jinli_lme.yaml configuration - Add structured output validation in session review step - Include detailed metadata reporting with session statistics and errors * fix(benchmark): adjust default concurrency for auto_memory job - Changed default concurrency from 3 to 1 for auto_memory job to prevent API overload - Updated help text to reflect new default value of 1 for concurrency parameter - Modified documentation to clarify concurrency behavior varies by job type * refactor(search): replace hardcoded candidate multiplier with constant - Introduced _CANDIDATE_MULTIPLIER constant set to 10 - Replaced hardcoded factor of 5 with _CANDIDATE_MULTIPLIER in BM25 search - Replaced hardcoded factor of 5 with _CANDIDATE_MULTIPLIER in vector search - Updated test to verify both search steps use ten times limit for candidates - Imported VectorSearchStep and Bm25SearchStep in test module - Added comprehensive test case for candidate count calculation logic * feat(lme): add data inspection error handling with fallback mechanism - Implemented non-retryable data inspection error markers detection - Added _is_data_inspection_error method to identify inspection failures - Created fallback handling for data inspection errors in auto memory extraction - Added fallback handling for data inspection errors in session review - Extended failed extracts tracking with non-retryable and fallback flags - Separated fallback extracts from regular failed extracts in reporting - Enhanced error logging with specific data inspection failure messages - Updated metrics to track fallback extractions and reviews separately - Maintained existing retry logic for other exception types * feat(benchmark): enhance session review statistics with fallback tracking - Add support for identifying and listing non-retryable fallback reviews - Introduce --list-fallback argument to display fallback review details - Separate retryable failures from non-retryable fallbacks in reporting - Track fallback samples and sessions separately from failed ones - Update console output to show both retryable and non-retryable categories - Include fallback details in JSON output with reasons and session info - Modify failure counting logic to distinguish between retryable and fallback reviews * feat(benchmark): add question_id tracking and enhanced fallback reporting - Add question_id function to extract query.question_id from data - Initialize question_id_by_id dictionary to store question IDs by index - Store question_id for each sample during data processing - Enhance fallback output to include question IDs and session information - Format sample labels with question IDs when available - Display session IDs associated with each fallback case * feat(benchmark): add question_id support and improve bad sample reporting - Add question_id_for function to extract question_id from multiple sources - Add sample_label function to format samples as idx(question_id) when available - Store question_id in data dictionary during processing - Change bad_golden and bad_sessions to store full records instead of just indices - Update list_bad output to show formatted labels with question_id information - Improve error reporting with more detailed sample identification * feat(benchmark): enhance golden check stats with structured output - Add related_session_ids function to extract session IDs from verdict records - Create grouped_records function to group records by question type - Replace flat list output with JSON-formatted grouped records in list_bad option - Replace flat list output with JSON-formatted grouped records in list_bad_sessions option - Maintain Chinese labels while adding structured data presentation - Improve readability of bad verdict record display with hierarchical grouping * feat(benchmark): update data structure for question indexing - Replace sample_label with _idx field for index tracking - Add question_id field to store _question_id values - Maintain backward compatibility with empty string defaults - Preserve existing session_id functionality - Update data mapping to include new fields in grouped results * refactor(benchmark): streamline golden answer verification process - Replace relevance filtering with comprehensive information extraction - Remove is_relevant field and simplify session summary structure - Change relevant_info to extracted_info for clarity - Update golden check logic to work with full extractions instead of filtered summaries - Simplify prompt instructions to focus on complete information extraction - Remove redundant schema validation and structured output requirements - Adjust statistics calculation to match new extraction approach - Update metadata field names to reflect extraction rather than relevance checking * feat(benchmark): add selective file deletion option to clean_sample_outputs - Add --filename argument to delete only specific root-level files - Modify iter_sample_targets function to accept optional filenames filter - Implement validation for root-level filename constraints - Update function calls to pass filenames parameter - Add example usage for selective file deletion in documentation * feat(benchmark): add error count metrics to golden check statistics - Added golden_bad, session_bad, and both_bad calculation fields - Updated console output format to include error counts per question type - Modified table display to show both accuracy rates and error numbers - Enhanced statistical summary with additional error breakdown metrics * test(search): update search step tests with include_source parameter - Added include_source=False parameter to VectorSearchStep initialization - Added include_source=False parameter to Bm25SearchStep initialization - Maintained existing RuntimeContext parameters for both search steps - Updated test calls to match new constructor signature with include_source option |
||
|
jinliyl
|
90e7adc2d2
|
chore(config): disable embeddings by default and update documentation (#341)
- Set default version to 0.4.1.0 - Comment out embedding configuration in default.yaml - Update README and README_ZH to clarify embedding components are disabled by default - Add note explaining how to enable embedding-based semantic retrieval - Adjust table formatting and descriptions in documentation - Modify search command description to reflect vector search availability when enabled |
||
|
Sen Huang
|
6a2dd02e48
|
docs: restructure documentation and update content organization (#339)
* docs: restructure documentation and update content organization * docs: update documentation structure and add application scenarios |
||
|
jinliyl
|
b1c9bf67bf
|
fix(embedding): make input truncation CJK-aware (#337)
* fix(embedding): make input truncation CJK-aware * test(embedding): cover CJK-aware truncation budget |
||
|
Ziyang Guo
|
e41b1673ad
|
fix(search): honor min_score in plain search steps (#338) | ||
|
Ziyang Guo
|
c5eefe4da3
|
fix(search): expose markdown frontmatter on chunks (#314)
* fix(search): expose markdown frontmatter on chunks * style(search): apply pre-commit formatting * fix(search): make frontmatter chunk metadata opt-in * fixup! fix(search): expose markdown frontmatter on chunks * feat(markdown): add include_frontmatter_keys_in_metadata allow-list opt-in --------- Co-authored-by: RerankerGuo <1875366113@qq.com> Co-authored-by: Ziyang Guo <121015044+RunMarshal@users.noreply.github.com> |
||
|
jinliyl
|
2612d25959
|
feat(lme): add cli execution and agentic search tooling (#334)
* feat(service): add CLI service for local job execution - Introduce CliService to execute single jobs locally without serving ports - Add prepare_start_config and should_precheck_start functions for CLI job setup - Update reme start command to use CLI service when job argument is provided - Change default service backend from http to cli in jinli_lme config - Modify SearchStep to use constants and rename configuration parameters - Add unit tests for CLI service functionality and configuration handling - Update file extension support to include json format in addition to md and jsonl * feat(search): add BM25 and vector search steps with configuration updates - Add Bm25SearchStep and VectorSearchStep classes with tool context deduplication - Register new search step components in index module - Update configuration to use separate vector_search and bm25_search endpoints - Modify LLM models from qwen3.7-plus/glm-5.1 to glm-5.2 variants - Adjust search parameters and remove hybrid search implementation - Configure embedding store as default in storage settings - Remove auto-memory and file catalog configurations - Update watch directories from multiple paths to session_dir only * feat(agent): add tool result offloading and workspace management - Add tool_results_dir configuration option for offloaded tool results storage - Implement ToolResultOffloadMiddleware to persist large tool results to files - Create WorkspaceBackend to standardize file operations across tools - Add configurable builtin tools selection with sequential execution option - Integrate middleware support for agent wrapper with offloading capability - Update application initialization to create tool results directory - Add safety mechanisms for filesystem operations with sanitized filenames - Enhance agent wrapper with configurable working directory handling - Upgrade agentscope dependency to version 2.0.4 for improved features # Conflicts: # reme/application.py * feat(benchmark): add LongMemEval agentic search and result management - Introduce AgenticAnswerStep for agent-based history search - Add LmePrepareJudgeStep and LmeSaveResultStep for evaluation pipeline - Implement AddDraftStep and ReadAllDraftStep for evidence accumulation - Update configuration with new agent wrapper and search parameters - Add comparison script for analyzing agent run differences - Include documentation for LongMemEval failure analysis - Enhance tool result offloading with skip options - Modify search defaults and indexing behavior * feat(agent): implement tool result offloading with system reminders - Added tool_result_offload_message parameter to agent wrapper reply method - Implemented configurable reminder template for offloaded tool results - Created system reminder messages when tool results are offloaded to files - Added Chinese user message template for agentic answer step - Updated tool result offloading middleware to use custom reminder templates - Enhanced agentic answer instructions to handle long tool results via draft storage * feat(scripts): add LongMemEval results summarization tool - Create summarize_lme_results.py script to analyze result JSON files - Implement command line interface with answer id and dataset root options - Add support for specifying index range with start and end parameters - Include option to show failure details and non-successful completions - Calculate completion statistics and accuracy metrics - Display detailed breakdown of yes/no/other judgements - Handle missing and unreadable result files gracefully - Format output with percentages and comprehensive summary statistics * feat(summarize_lme_results): add question type breakdown to result summary - Import defaultdict from collections module - Add by_type dictionary to track statistics by question type - Count completed, yes, no, and other responses for each question type - Display detailed breakdown table showing accuracy by question type - Include question type column when processing judgements - Print comprehensive summary with question type distribution - Calculate and display accuracy percentage for each question type category * feat(lme): switch to qwen3.7-max model and add shuffle functionality - Changed default LLM model from glm-5.1 to qwen3.7-max in jinli_lme.yaml - Added random module import for shuffle functionality - Implemented --shuffle argument with BooleanOptionalAction for dataset shuffling - Added --seed argument to control random seed for reproducible shuffling - Applied random shuffle to dataset indices when shuffle is enabled - Added console output showing shuffle operation and seed information * fix(cli): set default random seed for shuffle functionality - Changed default seed value from None to 42 for consistent shuffling behavior - Ensures reproducible results when using shuffle option without explicit seed - Maintains backward compatibility while providing deterministic defaults * refactor(benchmark): update agentic answer guidelines for grounding - Updated English instruction to emphasize strict grounding in retrieved context - Modified Chinese instruction to stress evidence-based responses without inference - Removed redundant conciseness requirement in both language versions - Enhanced clarity on proper use of draft saving and retrieval mechanisms - Strengthened emphasis against hallucination of unsupported facts * refactor(benchmark): update agentic search instructions and configuration - Replace separate vector_search and bm25_search with unified search tool - Update agent instructions to use single search tool with multiple strategies - Simplify Chinese instructions for search methodology - Add comprehensive search tool configuration with hybrid vector/BM25 capabilities - Increase model retry attempts from 1 to 3 for better reliability - Remove redundant tool references from job_tools list * feat(search): add configurable search limit with environment variable support - Remove hardcoded limit and min_score parameters from config schema - Increase LLM context size from 200000 to 1000000 - Add REME_SEARCH_LIMIT environment variable support for search configuration - Implement command line argument --search-limit to override default search limit - Add input validation to ensure search limit is positive - Modify subprocess execution to pass environment variables - Update search step to use dynamic default limit from environment or fallback to 5 * refactor(benchmark): remove agentic answer step and related configurations - Removed AgenticAnswerStep class and its registration - Deleted agentic_answer.yaml prompt configuration file - Removed agentic answer related job definitions from jinli_lme.yaml - Cleaned up tool result offloading middleware implementation - Removed tool_results_dir configuration field from application config - Deleted comparison and analysis scripts for agent runs - Removed agentic answer step from LME init module exports - Updated agent wrapper to remove tool result offloading functionality - Removed unused imports and dependencies in agent wrapper module * refactor(benchmark): remove unused LME result processing components - Removed LmePrepareJudgeStep and LmeSaveResultStep classes from benchmark module - Cleaned up imports and exports in lme module initialization - Removed unused middleware configuration from agent wrapper - Deleted obsolete result.py file containing deprecated result processing logic - Simplified agent instantiation by removing middleware parameter - Updated import statements to reflect removed dependencies * refactor(index): remove unused search steps and update imports - Remove Bm25SearchStep and VectorSearchStep from index steps module - Remove unused prepare_start_config and should_precheck_start exports - Move import statements to proper location in reme.py - Update test module to use direct import path for CliService - Remove vector_search and bm25_search configurations from jinli_lme.yaml - Add workspace directory environment variable configuration - Add docstring to getcwd method in agent wrapper - Remove empty middleware list from agent wrapper initialization * feat(index): add BM25 and vector search steps with tool context deduplication - Add Bm25SearchStep for plain BM25 keyword search with tool_context deduplication - Add VectorSearchStep for plain vector search with tool_context deduplication - Implement tool context state management with TTL-based deduplication - Add support for chunk deduplication across tool contexts within TTL window - Update index steps module to include new search step classes - Add test coverage for CLI metadata output functionality - Refactor CLI service to remove unused show_status parameter - Update documentation comments to reflect internal service configuration * feat(steps): add Python code execution capability - Introduce PythonExecuteStep to run Python code in subprocess - Add configuration for python_execute step in jinli_lme.yaml - Register python_execute in available tools list - Implement timeout handling with default 60 second limit - Capture stdout/stderr output and return code metadata - Add comprehensive unit tests for execution scenarios - Support workspace directory context for code execution - Handle timeout errors and runtime exceptions gracefully * refactor(python_execute): replace subprocess with asyncio for Python code execution - Replace subprocess.run with asyncio.create_subprocess_exec for non-blocking execution - Add _PythonResult dataclass to encapsulate execution results and timeout status - Implement proper timeout handling with asyncio.wait_for and process.kill() - Update metadata to include returncode and stderr when timeout occurs - Convert synchronous _run_python method to asynchronous implementation - Maintain backward compatibility while improving execution reliability * refactor(python_execute): replace subprocess with asyncio for Python code execution - Replace subprocess.run with asyncio.create_subprocess_exec for non-blocking execution - Add _PythonResult dataclass to encapsulate execution results and timeout status - Implement proper timeout handling with asyncio.wait_for and process.kill() - Update metadata to include returncode and stderr when timeout occurs - Convert synchronous _run_python method to asynchronous implementation - Maintain backward compatibility while improving execution reliability |
||
|
xyf2020
|
82971ac5b0
|
feat(chunker): better json chunker and json chunker (#325)
* feat(file_chunker): add dedicated JSON and JSONL file chunkers - Add JsonFileChunker: structure-aware chunking preserving nested key paths, optional list-to-dict conversion, size measured by json.dumps() char count - Add JsonlFileChunker: line-aligned sliding-window chunking with configurable overlap, supports char/byte mode switching - Register both chunkers in default.yaml (json for .json, jsonl for .jsonl) - Add comprehensive unit tests (21 + 20 test cases) * chore(config): update default chunker supported_extensions to txt/log * refactor(json_chunker): optimize _build_tree O(n²) serialization and rewrite tests - Fix O(n²) redundant json.dumps in _build_tree: * Empty containers handled directly as leaves (0 serialization) * Non-empty containers recurse first, then reconstruct+dump once * Only containers that become leaves pay serialization cost - Add _reconstruct_object/_reconstruct_array helpers - Remove dead code: _merge_json method - Apply user changes: min_element_size formula 0.01->0.05, threshold < to <= - Use indent=None for compact output (consistent with _SizeNode estimation) - Remove unused _text_size from JsonlFileChunker Test rewrite: - Replace try/finally boilerplate with make_json fixture - Group tests into TestXxx classes with pytest.mark.parametrize - Add TestOutputValidation: 9 parametrized scenarios verifying: * All chunks are valid JSON * Text length <= chunk_chars (with single-leaf tolerance) * Leaf-value concatenation matches original data (dict + array roots) - Add TestSizeNode: incremental size accuracy tests - Add TestDfsAlgorithm: path wrapping, DFS order, calibration tests - Update test_min_element_size_formula for new 0.05 multiplier - Update test_build_tree_structure for larger min_element_size * chore: apply black formatting to test files |
||
|
jinliyl
|
41c6cdaff5
|
Bump version to 0.4.0.9 | ||
|
jinliyl
|
b53d3db8d0
|
chore(workflow): update package installation to include core extra de… (#331)
* chore(workflow): update package installation to include core extra dependencies - Modified pre-commit workflow to install with [dev,core] extras - Updated python-publish workflow to install wheel with core extra dependency - Changed from direct dist/*.whl install to variable assignment for wheel path - Ensured core dependencies are included during test installation phase * chore(workflow): remove docs deployment workflow - Delete the entire docs.yml workflow file that was used for deploying documentation - Remove all related configuration including build and deploy jobs - Stop automatic deployment of docs on pushes to main branch - Remove GitHub Actions workflow for docs/ directory changes |
||
|
jinliyl
|
eb471d7d94
|
fix(embedding): reject mismatched embedding dimensions (#330)
* fix(embedding): enforce strict dimension matching for embeddings - Add _embedding_dim_matches method to validate embedding dimensions - Reject embeddings with mismatched dimensions instead of padding/truncating - Drop stale embeddings with wrong dimensions during loading and upsert operations - Disable embedding store when query dimensions don't match configured dimensions - Fail health checks when embedding dimensions don't match expected values - Skip chunks with wrong dimensions during FAISS index rebuild - Add comprehensive tests for dimension validation behavior * refactor(file_store): simplify conditional checks in vector search and test assertions - Combine multiple conditionals into single check for empty FAISS index - Replace explicit empty list comparison with boolean check for node embedding calls - Maintain same functional behavior while improving code readability * fix(embedding): harden dimension validation helpers |
||
|
jinliyl
|
38cf16071b
|
refactor(embedding): update embedding model initialization and session storage paths (#329)
* refactor(embedding): update embedding model initialization and session storage paths - Remove unused inspect import from as_embedding module - Pass dimensions directly to embedding model constructor instead of using parameters - Update session state file paths to use mem_session directory instead of resource - Add mem_session_dir configuration option to application config schema - Update workspace directory creation to include new mem_session directory - Change AgentScope and Claude Code session paths to use mem_session directory - Move embedding dimensions from parameters to top-level configuration - Update AgentScope dependency version from 2.0.3 to 2.0.4 - Update integration tests to reflect new session file location paths * chore(version): bump version to 0.4.0.8 - Update __version__ from 0.4.0.7 to 0.4.0.8 in __init__.py |
||
|
jinliyl
|
10da205797
|
feat(benchmark): add lme benchmark steps (#326)
* refactor(search): replace time module with datetime for timestamp generation - Removed unused time import - Added static method _now_ts using datetime.timestamp - Updated clock parameter to use _now_ts method instead of time.time - Maintained same timestamp precision and functionality * test(http): add tests for HTTP client display formatting - Add test for default metadata hiding behavior in CLI output - Add test for metadata display when show_metadata is enabled - Verify _format_for_display method correctly formats response text - Test both success case and metadata inclusion scenarios * chore(build): remove longmemeval from gitignore - Removed longmemeval directory from gitignore list - Kept evaluation and datasets directories in ignore list - Updated gitignore configuration for proper version control |
||
|
jinliyl
|
bf902b3479
|
Bump version to 0.4.0.7 | ||
|
Sen Huang
|
0a7eea18f8
|
fix(as_embedding): support both agentscope 2.0.2 and 2.0.3 (#323)
2.0.3 promoted `dimensions` to a required first-class constructor argument while keeping a backfill from `parameters.dimensions`; 2.0.2 has no such argument and reads `dimensions` from `Parameters`. Keep `dimensions` in `Parameters` for both versions and, when the model constructor accepts `dimensions`, pass `dimensions=None` so 2.0.3's backfill promotes it out of `parameters`. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
imrewce
|
1e798d3b4e
|
fix(file_io): fix risk of out-workspace paths (#322)
* fix(file_io): fix risk of out-workspace paths * chore(file_io): remove unused unittest file |
||
|
jinliyl
|
7369342115
|
feat(search): add tool context deduplication and improve search configuration (#321)
* feat(search): add tool context deduplication and improve search configuration - Modify _make_tool methods to accept and inject tool_context_id parameter - Add tool_context_id handling in AS and CC agent wrappers - Increase search candidate multiplier from 3.0 to 5.0 in default config - Extend HTTP client timeout from 30s to 3600s - Add tool context deduplication logic to prevent duplicate search results - Implement TTL-based expiration for seen chunks in tool contexts - Add comprehensive unit tests for tool context deduplication behavior - Update .gitignore to exclude longmemeval directory - Add time import for timestamp functionality in search step * refactor(search): replace time module with datetime for timestamp generation - Removed unused time import - Added static method _now_ts using datetime.timestamp - Updated clock parameter to use _now_ts method instead of time.time - Maintained same timestamp precision and functionality |
||
|
xyf2020
|
43a407bc4f
|
feat: add start_date/end_date time filter support for search job (#317)
* feat: add start_date/end_date time filter support for search job
- Add _extract_date_from_path to extract validated YYYY-MM-DD from chunk paths
- Add start_date/end_date filtering in _matches_search_filter
- Implement progressive recall in FaissLocalFileStore.vector_search
- Promote start_date/end_date from context to search_filter in SearchStep
- Add start_date/end_date parameters to search job in default.yaml
- Add unit tests for date filter functionality
* fix: validate/normalize date filters and harden _extract_date_from_path
Address three code-review comments on the time_filter search feature:
1. Validate/normalize start_date and end_date before string comparison.
_matches_search_filter does lexicographic comparison against path_date
(always canonical YYYY-MM-DD). Raw caller values like '2026-2-28' or
'abc' would produce silently wrong results. Now SearchStep normalizes
valid dates via extract_daily_date (with strptime fallback for
non-zero-padded input) and silently ignores invalid dates with a
logger.warning, removing them from the filter.
2. Clarify behavior for paths without embedded dates.
Added optional strict_date_filter parameter (default False). When True
and at least one date bound is active, chunks whose path yields no date
(e.g. digest/personal/topic.md) are excluded. When False (default),
the existing behavior is preserved — dateless paths pass through.
3. Harden _extract_date_from_path against non-standard suffixes.
Previously parts[1].split('.')[0] accepted '2026-05-18.anything' as a
valid date. Now only exact 'YYYY-MM-DD' (dir) and 'YYYY-MM-DD.md'
(day-index) forms are accepted.
|
||
|
Zhaoyang Liu
|
f63165c66b
|
update the readme, reorg the content (#318) | ||
|
jinliyl
|
6bf2db8ff4
|
Update agentscope dependency version to 2.0.3 | ||
|
Sen Huang
|
8877743ca9
|
feat(cli): route bare commands to the running server's real config (#312)
Some checks failed
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Deploy Docs / deploy (push) Has been cancelled
Deploy Docs / build (push) Has been cancelled
call_server now resolves the client backend/transport/host/port from the live `reme start` process (replaying its start args through resolve_app_config) so a bare `reme <action>` reaches the server however it was actually launched, falling back to local config when none runs. Explicit backend=/transport=/host=/port= still win. Also fix as_embedding to pass `dimensions` explicitly for agentscope >=2.0.2, and add Claude Code auto-memory/auto-dream demos to the READMEs. |
||
|
Ziyang Guo
|
c060933e4d
|
fix(auto-memory): preserve message timestamps (#310)
* fix(auto-memory): preserve message timestamps * fix(auto-memory): infer daily date from messages * feat(file_io): add strict date parsing and improve daily date handling - Add new parse_daily_date function for strict YYYY-MM-DD validation - Replace extract_daily_date with parse_daily_date for explicit date validation - Change _messages_day to use max date instead of min for historical imports - Reorder imports to maintain consistent module ordering - Move session message saving after date validation in auto_memory - Add comprehensive tests for invalid date rejection before saving - Add tests for strict YYYY-MM-DD date format validation - Update test names to reflect latest date behavior --------- Co-authored-by: Ziyang Guo <121015044+RunMarshal@users.noreply.github.com> Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com> |
||
|
Sen Huang
|
5a3450ddb3
|
chore(release): bump version to 0.4.0.6 (#309) | ||
|
Sen Huang
|
435aa713a2
|
fix(config): correct indentation in default.yaml (#308) | ||
|
Ziyang Guo
|
9d14e988d8
|
docs(framework): clarify context management boundary (#306)
Co-authored-by: Ziyang Guo <121015044+RunMarshal@users.noreply.github.com> |
||
|
jinliyl
|
1c05d0359b
|
feat(README): Enhance documentation styling, content, and layout (#304)
Some checks are pending
Deploy Docs / build (push) Waiting to run
Deploy Docs / deploy (push) Blocked by required conditions
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
* style(docs): update visual styling and layout of documentation figures - Change background color from #f7f8fb to #fffdf8 - Update fonts to include Comic Sans MS and Bradley Hand for titles - Adjust stroke colors and widths for panels and chips - Modify arrow and line styles with new colors and dimensions - Update marker sizes and colors for better visual consistency - Add rounded corners and join styles for smoother appearance - Apply dashed borders to chip elements - Refine color palette for text and UI elements docs(readme): enhance documentation content and formatting - Improve readability with better line breaks and spacing - Update core ideas section with expanded descriptions - Add news section announcing ACL 2026 paper acceptance - Enhance agent integration section with detailed examples - Revise automatic memory flow description for clarity - Update workspace operation interface with improved categorization - Standardize table formatting and column widths - Clarify directory structure with better organization - Add minimal CLI examples for easier integration - rename session_id to session_event in directory structure - add example files under digest directory structure * style(docs): adjust image dimensions in README table - Changed table cell widths from 45% to 50% for better alignment - Reduced image width from 100% to 92% to prevent overflow - Applied consistent sizing across all four documentation images - Improved visual balance of the feature comparison table * docs(readme): update documentation with design philosophy and operation interface - Change background color in design-philosophy.svg from #f7f8fb to #ffffff - Rename 'Workspace Operation Interface' to 'ReMe Operations' in README.md - Update Chinese documentation with consistent 'ReMe Operations' title - Adjust image widths from 100% to 92% in Chinese documentation tables - Standardize summary titles in both English and Chinese documentation * docs(readme): add demo videos for auto memory and auto dream features - Added expandable details section with video demonstrations - Included side-by-side comparison of Auto Memory and Auto Dream features - Added video controls with muted loop and inline playback support - Updated both English and Chinese README files with identical content - Used table layout for proper alignment of demonstration videos - Maintained consistent styling and formatting across both language versions * docs(figure): remove qwenpaw auto memory video file - Delete the video file qwenpaw-auto-memory.mp4 from docs/figure directory - Remove all video content related to auto memory demonstration - Clean up media assets that are no longer needed in documentation * style(docs): replace details summary with centered paragraph in README files - Replaced collapsible details/summary elements with centered paragraphs - Removed unnecessary br tags in both English and Chinese documentation - Maintained the same visual presentation while simplifying HTML structure - Updated both README.md and README_ZH.md consistently * style(docs): update design philosophy diagram styling - Changed fonts to include Comic Sans MS and Bradley Hand for titles and labels - Updated color scheme with darker text colors (#1f2430 instead of #172033) - Increased stroke widths from 1.2 to 2.2 for panels and adjusted other stroke values - Added rounded line caps and joins for smoother visual appearance - Modified chip styling with dashed borders and updated stroke properties - Adjusted arrow markers to smaller sizes with updated dimensions - Refined color values for arrows, links and file lines for better contrast - Applied consistent stroke properties across all visual elements * docs(readme): update documentation and adjust svg dimensions - Updated SVG canvas dimensions from 640px to 670px height - Simplified Skill + CLI integration examples in README tables - Removed detailed command examples and collapsible sections - Streamlined automatic memory capabilities documentation - Cleaned up ReMe operations table formatting - Consolidated command usage instructions for clarity |
||
|
jinliyl
|
6244e7eeaa
|
feat(README): Enhance documentation styling, content, and layout (#303)
* style(docs): update visual styling and layout of documentation figures - Change background color from #f7f8fb to #fffdf8 - Update fonts to include Comic Sans MS and Bradley Hand for titles - Adjust stroke colors and widths for panels and chips - Modify arrow and line styles with new colors and dimensions - Update marker sizes and colors for better visual consistency - Add rounded corners and join styles for smoother appearance - Apply dashed borders to chip elements - Refine color palette for text and UI elements docs(readme): enhance documentation content and formatting - Improve readability with better line breaks and spacing - Update core ideas section with expanded descriptions - Add news section announcing ACL 2026 paper acceptance - Enhance agent integration section with detailed examples - Revise automatic memory flow description for clarity - Update workspace operation interface with improved categorization - Standardize table formatting and column widths - Clarify directory structure with better organization - Add minimal CLI examples for easier integration - rename session_id to session_event in directory structure - add example files under digest directory structure * style(docs): adjust image dimensions in README table - Changed table cell widths from 45% to 50% for better alignment - Reduced image width from 100% to 92% to prevent overflow - Applied consistent sizing across all four documentation images - Improved visual balance of the feature comparison table * docs(readme): update documentation with design philosophy and operation interface - Change background color in design-philosophy.svg from #f7f8fb to #ffffff - Rename 'Workspace Operation Interface' to 'ReMe Operations' in README.md - Update Chinese documentation with consistent 'ReMe Operations' title - Adjust image widths from 100% to 92% in Chinese documentation tables - Standardize summary titles in both English and Chinese documentation * docs(readme): add demo videos for auto memory and auto dream features - Added expandable details section with video demonstrations - Included side-by-side comparison of Auto Memory and Auto Dream features - Added video controls with muted loop and inline playback support - Updated both English and Chinese README files with identical content - Used table layout for proper alignment of demonstration videos - Maintained consistent styling and formatting across both language versions * docs(figure): remove qwenpaw auto memory video file - Delete the video file qwenpaw-auto-memory.mp4 from docs/figure directory - Remove all video content related to auto memory demonstration - Clean up media assets that are no longer needed in documentation * style(docs): replace details summary with centered paragraph in README files - Replaced collapsible details/summary elements with centered paragraphs - Removed unnecessary br tags in both English and Chinese documentation - Maintained the same visual presentation while simplifying HTML structure - Updated both README.md and README_ZH.md consistently * style(docs): update design philosophy diagram styling - Changed fonts to include Comic Sans MS and Bradley Hand for titles and labels - Updated color scheme with darker text colors (#1f2430 instead of #172033) - Increased stroke widths from 1.2 to 2.2 for panels and adjusted other stroke values - Added rounded line caps and joins for smoother visual appearance - Modified chip styling with dashed borders and updated stroke properties - Adjusted arrow markers to smaller sizes with updated dimensions - Refined color values for arrows, links and file lines for better contrast - Applied consistent stroke properties across all visual elements |
||
|
Sen Huang
|
e7ef2c8ce6
|
feat(docs): add multilingual documentation with GitHub Pages deployment (#287)
Some checks are pending
Deploy Docs / build (push) Waiting to run
Deploy Docs / deploy (push) Blocked by required conditions
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
* feat(docs): add multilingual documentation with GitHub Pages deployment * docs(readme): update agent integration documentation with current status |
||
|
Sen Huang
|
be7d1c0cf2
|
refactor(transfer): drop orphaned ingest step, make service discovery cross-platform (#300)
* refactor(transfer): drop orphaned ingest step, make service discovery cross-platform - Remove ingest step: superseded by auto_resource (drop files under resource/ → watcher interprets them); its meta.json/<date>.md outputs had no consumers and tripped the auto_resource watcher. - Replace lsof/pgrep shell-outs in service_utils with psutil (per-process enumeration, no root needed on macOS) for Windows/macOS/Linux support. - Add cross-platform test coverage for _pid_on_port / _scan_reme_procs. - Deps: +psutil, -filelock (only used by the removed ingest lock). * chore(release): bump version to 0.4.0.5 |
||
|
Sen Huang
|
3dee10d4f9
|
feat: add Claude Code plugin with auto-memory functionality (#297)
* feat: add Claude Code plugin with auto-memory functionality * refactor(auto_memory): fix spacing in json parsing logic |
||
|
jinliyl
|
ad7893e9c4
|
fix(mcp): resolve circular import issues and update dependencies (#296)
* fix(mcp): resolve circular import issues and update dependencies - Moved fastmcp imports inside functions to prevent circular dependencies - Replaced _TRANSPORT_MAP with _VALID_TRANSPORTS set for transport validation - Updated version number from 0.4.0.3 to 0.4.0.4 - Added claude-agent-sdk dependency to core optional dependencies - Used TYPE_CHECKING imports for FastMCP related types - Restructured transport mapping logic within function scope - Fixed string annotation for CallToolResult type hints * refactor(tests): update date handling in daily steps tests - Replace _date.today() with timezone-aware now function - Use Asia/Shanghai timezone for date formatting - Change return format to use strftime instead of isoformat - Import now function from reme.steps.evolve module * refactor(tests): clean up unused imports in daily steps test - Removed unused date import from datetime module - Removed redundant pathlib Path import that was already imported later - Kept necessary imports for asyncio, os, tempfile, warnings, and frontmatter modules * test(daily_steps): update test to include application context for daily list step - Add ApplicationContext initialization with temporary workspace directory - Register file store component in application context - Pass application context to DailyListStep constructor - Maintain existing test assertion behavior for date metadata verification |
||
|
jinliyl
|
ffb4d08c4f
|
feat(mem): Enhance daily note system with metadata handling and write functionality (#295)
* feat(file_io): add daily_write step for creating daily notes with conversation metadata - Add DailyWriteStep class that delegates to write job for creating daily notes - Register daily_write job in default configuration with proper parameters - Include validation for name and session_id path components - Add test coverage for daily_write functionality including metadata handling - Preserve existing job execution method in application.py after repositioning - Update base_step.py to use positional-only parameter syntax for job methods - Import and expose DailyWriteStep in file_io module initialization - Override reserved metadata keys (name, description, session_id, source_conversation) with fixed values - Refresh daily index after successful write operation - Generate proper source conversation links in markdown format * feat(daily): refactor daily note system with enhanced metadata handling - Introduce validate_filename_component function and export it - Add _INDEX_HIDDEN_METADATA_KEYS to hide conversation metadata from index - Update scan_notes to exclude hidden metadata keys from index rendering - Modify auto_memory to use daily_write tool and manage session frontmatter - Implement session note lookup and renaming based on frontmatter name - Update daily_list to return flattened note metadata including session info - Change daily_write to dispatch write step instead of running job - Add test cases for updated daily note functionality and metadata handling - Update version from 0.4.0.2 to 0.4.0.3 * fix(evolve): correct metadata update in auto memory response - Fixed trailing comma issue in metadata dictionary update - Ensured proper formatting of response metadata structure - Maintained existing functionality while fixing syntax error * refactor(auto_resource): replace daily_create with dynamic note management - Remove DailyCreateStep and related exports from file_io module - Replace static daily note creation with dynamic resource-linked card system - Implement LLM-suggested naming with frontmatter-driven file management - Add source_resource linking for tracking original files - Introduce collision handling with hash-based suffixes - Update documentation to reflect new resource card workflow - Modify auto_resource prompts to use write/edit tools instead of daily_create - Adjust test fixture comments to match new agent behavior - Update framework diagrams and quick start examples accordingly * feat(app): add version info to app initialization and update auto-memory logic - Include version number in application startup logging - Remove tool result truncation logic from auto-memory step - Update auto-memory to exclude tool_result blocks from saved history - Add test case to verify tool results are filtered out from message saving - Update YAML prompts to clarify filename naming rules without dates - Modify configuration to support new dispatch steps format with persistence control * feat(auto_memory): add note modification tracking and optimize frontmatter updates - Add _note_bytes and _note_modified methods to track actual file changes - Optimize frontmatter updates by checking existing metadata before update - Add modified flag to response metadata indicating actual note changes - Update logging to include modified status in various operations - Add comprehensive tests for modified/unmodified detection scenarios - Enhance result hook logic to skip when no actual changes occur - Refactor metadata handling to properly track creation vs modification status |
||
|
Sen Huang
|
a3ea4d2622
|
docs(logo): update reme logo image (#294) | ||
|
jinliyl
|
8b82ff88d0
|
feat(evolve): enhance agent reply processing and logging capabilities (#293)
* feat(evolve): enhance agent reply processing and logging capabilities - Add agent_reply_result_text function to extract final user-visible text from agent replies - Implement comprehensive logging throughout auto_memory, auto_resource, and dream modules - Add max_units configuration option to limit extracted memory units - Improve error handling and validation in auto_resource step - Refactor dream extract step to respect max_units limit during processing - Enhance summary rendering in dream finish step with detailed breakdown - Add result hook functionality for embedding hosts integration - Implement loose resource filename handling for root-level resources - Update test cases to reflect new functionality and improved error messages * test(background-steps): update fake upsert function to include created parameter - Modified fake_upsert function to accept 'created' parameter instead of '_created' - Added 'created' field to captured dictionary in fake_upsert function - Included 'created': True in the expected response dictionary for test case - Updated test assertion to match new parameter structure |
||
|
jinliyl
|
afe12b16db
|
feat(file_store): add embedding backfill for persisted chunks (#292)
- Implement _backfill_missing_embeddings method to handle chunks without embeddings - Add logic to identify and process chunks that predate embedding feature - Integrate backfill process into store loading sequence - Add proper error handling and logging for backfill operations - Create unit test for embedding backfill functionality - Ensure backfilled embeddings are properly persisted to storage |
||
|
jinliyl
|
7d86658f33
|
Refactor logging levels and add dream schema definitions (#291)
* chore(logging): change info logs to debug level for data loading operations - Changed stopwords loading log from info to debug level - Changed file catalog nodes loading log from info to debug level - Changed file graph nodes loading log from info to debug level * feat(dream): add dream schema definitions and enum for auto-dream functionality - Add DreamBucketEnum with procedure, personal, and wiki values - Create comprehensive dream-related Pydantic models including DreamUnit, DreamTopic, DreamExtractOutput, IntegrateOutcome, TopicSelectionOutput, ProactiveResult, and DreamState - Move schema definitions from local step module to shared schema package - Update dream extraction and integration steps to use new enum-based bucket validation - Initialize digest directories for each dream bucket type - Enhance embedding store health check with workspace directory logging * refactor(tests): update DreamState import path in test_auto_dream.py - Move DreamState import from reme.steps.evolve.dream.schema to reme.schema - Maintain same functionality with updated module reference - Align import with new schema location in project structure |
||
|
jinliyl
|
a3bd81bde2
|
Update version to 0.4.0.2 and improve tokenizer index handling (#290)
* fix(core): update version number to 0.4.0.1 - Incremented version from 0.4.0.0 to 0.4.0.1 in __init__.py * fix(index): remove stopwords path from tokenizer config and add keyword index repair - Remove stopwords_path from tokenizer config to prevent index forking by install path - Add _sync_keyword_index_from_chunks method to repair keyword index when persisted state mismatches - Implement test for keyword index repair from persisted chunks when missing - Add test to verify tokenizer fingerprint ignores stopwords absolute path - Update version from 0.4.0.1 to 0.4.0.2 * feat(dream): add scan_days parameter to dream extraction process - Add scan_days configuration option to default.yaml with default value of 2 - Implement recent_dates utility function to calculate date ranges for scanning - Modify DreamExtractStep to scan multiple days based on scan_days parameter - Update dream extraction to process files across multiple dates instead of single day - Extend DreamState schema to include dates and scan_days fields - Update DreamTopicsStep to handle multi-day topic processing - Modify finish step to checkpoint files from all scanned dates - Add comprehensive tests for multi-day scanning functionality - Update prompt templates to include scan dates information - Refactor topics writing logic to target specific date rather than current date |
||
|
jinliyl
|
8c1d348468
|
fix(core): update version number to 0.4.0.1 (#289)
- Incremented version from 0.4.0.0 to 0.4.0.1 in __init__.py |
||
|
Sen Huang
|
164b214b84
|
feat(tests): support .jsonl.zst files in integration tests (#288) | ||
|
Sen Huang
|
e31db5fe19
|
docs: rename vault_dir to workspace_dir in documentation and examples (#286)
* docs: rename vault_dir to workspace_dir in documentation and examples * refactor(extract): format long method call across multiple lines * refactor(extract): format system prompt parameters for better readability |
||
|
jinliyl
|
01a597aba4
|
chore(project): update package name from reme to reme-ai (#285)
- Changed project name in pyproject.toml from 'reme' to 'reme-ai' - Updated dependency references in full extras to use 'reme-ai[core]' and 'reme-ai[dev]' |
||
|
jinliyl
|
206a53e5ed
|
init: reme version 0.4.0 (#284) | ||
|
jinliyl
|
26cb5ca62f
|
Dev/0618 (#282)
* feat(core): enhance agent wrapper with session management and stream chunks
- Add AddStep for simple arithmetic operations
- Implement session persistence with AsStateHandler for AgentScope
- Introduce stream chunk conversion for unified event handling
- Add file session store for Claude Code agent backend
- Support skill integration and permission handling in agent wrappers
- Add proactive and daily topics features to auto dream step
- Refactor auto memory and auto resource steps to use job tools
- Update application shutdown sequence for proper resource cleanup
- Enhance environment loading utilities with parse_env_file function
- Add comprehensive error handling and validation for session IDs
* refactor(embedding): simplify model initialization and update component management
- Replace individual _start methods with shared model_cls attribute pattern
- Remove redundant _close methods from embedding model wrappers
- Add application-level component update capability via update_component method
- Simplify BaseAsLLM _start method with early return when model exists
- Remove unused skill instruction template from agent wrapper
- Restructure CronJob to execute its own steps instead of dispatching external jobs
- Update cron job configuration format to use steps instead of dispatch targets
- Refactor cron job tests to match new execution model
- Remove deprecated dispatch_step/dispatch_job functionality from cron job
* feat(agent): add session management and cleanup functionality
- Added session_dir configuration field for persisted agent sessions
- Included session directory in vault initialization process
- Implemented session retention period with configurable days
- Added automatic cleanup of expired session files
- Introduced session cleanup flag to prevent duplicate operations
- Updated project path calculation to use vault path directly
- Removed fallback streaming implementation from base class
* feat(agent): add session management and cleanup functionality
- Added session_dir configuration field for persisted agent sessions
- Included session directory in vault initialization process
- Implemented session retention period with configurable days
- Added automatic cleanup of expired session files
- Introduced session cleanup flag to prevent duplicate operations
- Updated project path calculation to use vault path directly
- Removed fallback streaming implementation from base class
* refactor(application): restructure job startup order and improve error handling
- Change job startup sequence from background-last to base-stream-background-cron
- Import specific job types (BackgroundJob, CronJob, StreamJob) instead of generic BaseJob
- Update isinstance checks for proper job type identification
- Implement robust error handling during component closure with preserved exceptions
- Modify job merging to combine config and call-time kwargs in BaseJob and StreamJob
- Update service job registration to return boolean success indicators
- Add timezone support for cron job scheduling
- Enhance keyword index persistence with component-specific filenames
- Add comprehensive file store consistency tests and search filtering capabilities
- Include tokenizer stopwords in package distribution
- Fix prompt handler validation behavior and error messages
* fix(core): handle exceptions during application startup and improve validation
- Add exception handling around component startup to close started components on failure
- Replace assertions with runtime checks in claim_channel step for Python -O compatibility
- Add input validation for config parser including empty keys and non-mapping roots
- Enhance environment variable expansion to convert scalar types
- Add support for relative config file paths by searching in config directory
- Validate 2D array requirements in batch cosine similarity function
- Update channel notify step to return proper response objects
- Pass client-specific arguments through CLI to HTTP client initialization
- Add comprehensive tests for error conditions and edge cases
* feat(graph): add Neo4j backend support with enhanced health monitoring
- Implement Neo4jFileGraph component with connection constraints and async operations
- Add cached count tracking for nodes, edges, and virtual nodes in Neo4j backend
- Update health check to include Neo4j status reporting with memory usage
- Modify LLM demo steps to always register add tool without conditional flag
- Enhance AddStep to handle numeric string conversion and input validation
- Add comprehensive unit tests for Neo4j integration and error handling scenarios
- Remove deprecated use_add_tool parameter from LLM demo components
- Update integration tests to reflect simplified tool registration approach
* feat(file_io): enhance file I/O operations with path validation and large file handling
- Add resolve_path function with comprehensive path validation and security checks
- Implement read_file_lines_safe for efficient reading of large files by line ranges
- Integrate path validation across all file I/O operations to prevent directory traversal
- Add proper error handling for invalid paths and file access issues
- Enhance daily index operations with path resolution and error reporting
- Add 'changed' field to index responses to track file modification status
- Update file listing operations to use secure path resolution
- Add support for JSONL files in default scanning operations
- Improve move and delete operations with proper path validation
- Add comprehensive path validation tests and security checks
* refactor(steps): update file I/O and prompt handling implementations
- Add module docstring to file_io/__init__.py
- Remove unused validate parameter from prompt_format method
- Update import path from reme.reme to reme4.reme in common_utils.py
- Add missing docstrings to test classes and methods
- Remove deprecated test_format_missing_variable_no_validate test
- Simplify assertion in test_job.py using not operator
- Update import statement in test_utils.py for common_utils
- Add docstrings to dummy classes and functions in tests
- Rename variable in get_node_embeddings for clarity
* refactor(steps): restructure step modules and update change handling
- Split monolithic steps module into channel, common, evolve, file_io, index, and transfer submodules
- Replace ScanStoreChangesStep and ScanCatalogChangesStep with unified InitChangesStep
- Remove ForeachDispatchStep and replace with direct dispatch_steps mechanism in InitChangesStep
- Update configuration to use new init_changes_step with dispatch_steps pattern
- Add coalesce_changes utility for collapsing duplicate file change events
- Enhance AutoResourceStep to handle batch changes instead of single file operations
- Introduce ClearStoreStep to replace ClearAndScanStep functionality
- Add async locks to LocalFileCatalog for thread-safe operations
- Update documentation to reflect new directory structure and session organization
* test(steps): add comprehensive unit tests for background steps and search functionality
- Add new test_background_steps.py with initialization and dispatch update tests
- Add test_index_update_loop_init_dispatch_updates_store_across_batches function
- Add test_digest_watch_loop_init_dispatch_updates_named_catalog_and_logs function
- Create new test_search_step.py with complete SearchStep unit test coverage
- Implement FakeSearchStore for isolated SearchStep testing without external dependencies
- Add hybrid search RRF merging test with vector and keyword result fusion
- Include keyword-only search test with min_score filtering functionality
- Add empty query validation test with early failure mechanism
- Test vector and keyword search method calls with proper parameter passing
- Verify score handling and result ranking in hybrid search scenarios
* refactor(file_io): remove session_agent prefix from daily note filenames
- Removed 'session_agent_' prefix from daily note file naming pattern
- Updated all references in auto_dream, auto_memory, auto_resource, and daily_steps
- Modified config documentation to reflect new filename pattern
- Changed session file storage location in auto_memory to reme_session/dialog/
- Added validate_session_id and write_file_safe imports to file_io module
- Updated tests to match new filename convention without 'session_agent_' prefix
- Fixed day index refresh logic to properly update note count descriptions
- Adjusted proactive step to use new file path pattern for session notes
* feat(auto_resource): change resource processing to use same-name daily notes
- Replace MD5-based session ID generation with UUID5 for agent sessions
- Compute note stem from resource filename instead of hashing for daily note naming
- Update delete handler to use note stem instead of session ID for file lookup
- Modify upsert handler to use note stem as session ID parameter
- Change execute method to require changes as list of dictionaries
- Update test cases to use changes array instead of individual file_path and change parameters
- Adjust test assertions to verify same-name daily note creation and modification
- Refactor session state storage to use AgentScope format and location
- Remove deprecated session_state file handling in favor of new note system
* docs(structure): update resource naming convention in documentation
- Change resource naming from hash-based to stem-based format
- Update file path references from resource_{hash(resource_name)}.md to {resource_stem}.md
- Modify documentation to reflect new resource storage structure
- Adjust auto-resource saving location to use resource stem instead of hashed name
* refactor(evolve): split auto_dream into multi-step pipeline with dedicated dream modules
- Replace single AutoDreamStep with 4-step pipeline: extract, integrate, topics, finish
- Create new dream module structure under reme4/steps/evolve/dream/
- Remove old dream.py, auto_dream.py, and daily_topics.py files
- Add DreamExtractStep, DreamFinishStep, DreamIntegrateStep, DreamTopicsStep, ProactiveStep
- Update evolve init to import new dream step classes instead of old modules
- Document complete auto_dream logic breakdown and refactoring plan in markdown
- Consolidate dream-related functionality into focused, testable components
- Maintain LLM integration while improving step separation and error handling
* refactor(dream): update system prompts and integration logic
- Replace 'Phase 1/2' terminology with descriptive agent names in prompts
- Update extraction and integration prompts to clarify unit processing flow
- Add explicit instructions about provenance tracking and wikilink handling
- Enhance validation for source material references in personal preferences
- Add fallback mechanism to preserve topics when selection fails
- Include punctuation handling guidelines for YAML parsing
- Add comprehensive tests for wikilink graph relationships
- Create new configuration file with complete service definitions
* refactor(steps): remove auto_dream and dream steps, update config access
- Removed auto_dream step implementation and its associated logic
- Removed dream step implementation including extract and integrate phases
- Replaced direct app_context.app_config access with config_value method
- Added config_value helper method to BaseStep for unified config access
- Updated auto_memory to use session_dir config and add source conversation links
- Modified auto_resource to use config_value for directory paths
- Updated daily_* steps to use config_value for daily directory
- Added exception raising in bm25_index save method on failure
- Introduced SOURCE_CONVERSATION_KEY constant for tracking source sessions
* refactor(embedding): update embedding component to use credential-based initialization
- Replace model_cls with credential_cls for authentication handling
- Update configuration structure to separate credential and parameters
- Modify health check to access model name through new attribute path
- Change input parameter from 'text' to 'inputs' for generic handling
- Add credential initialization and parameter parsing in _start method
- Update agentscope dependency to version 2.0.2
- Remove direct model class references in favor of credential-based lookup
|
||
|
jinliyl
|
83831ec90c
|
feat(core): enhance reme4 (#281)
### 1. Agent Wrapper（统一 Agent 后端抽象） - **`base_agent_wrapper.py`**：`reply()` 返回值从 `tuple[str, Any]` 改为 `dict`（含 `session_id` / `last_message` / `result` / 可选 `structured_output`）；`reply_stream()` 改为产出统一的 `StreamChunk`。废弃 `add_tools()`，改为 `add_job_tools(names: list[str])`（按名解析 BaseJob）与 `add_skills()`；新增 `_resolve_job_tools()`、`_merged_kwargs()`、`_chunk()` 辅助方法及 `project_path` / `project_skills_root` 属性。 - **`as_agent_wrapper.py`（AgentScope 后端）**： - 会话持久化重写：`session_path` 落地到 `<vault>/<session_dir>/agentscope/`，`_load_state` 支持 `resume` / `session_id` / `fork_session`，并做 UUID 校验（`_validate_session_id`）；`_cleanup_expired_sessions` 按天数清理过期会话。 - 新增内置工具集（`BypassAnalysisBash` + Edit/Glob/Grep/Read/Write），`BypassAnalysisBash` 绕过 AgentScope 自带 Bash 静态分析以让 permission_mode 生效；`_resolve_skills()` 把配置的 skill 暴露给后端，`_load_tool_env()` 注入项目 `.env`。 - `_event_to_chunk()` 把 20+ 种 AgentScope 事件（Reply/Text/Thinking/Data/ToolCall/ToolResult/ModelCall/ExceedMaxIters）归一化为 `StreamChunk`。 - **`cc_agent_wrapper.py`（Claude Code SDK 后端，+551 行）**： - 新增 `_CcFileSessionStore`：基于 vault 的文件型会话存储，实现 append（按 uuid 去重）/ load / list / delete / list_subkeys，并对路径做 `_safe_parts` + `resolve()` 防越界校验。 - `_build_options()`：统一构建 `ClaudeAgentOptions`，处理 skills、disallowed_tools（默认禁 `WebSearch`）、`.env` 注入、Claude Code 的 API 凭据解析（`_claude_code_api_env`，多级 base_url/api_key 回退）、`CLAUDE_CONFIG_DIR` 设置、skill 目录软链接（`_ensure_claude_skill_dir`）。 - `_raw_event_to_chunk()` / `_message_content_to_chunks()`：把 Anthropic 流式事件（message_start/delta/stop、content_block_*）与 SDK 消息块（AssistantMessage/UserMessage/ResultMessage/RateLimitEvent）转换为统一 `StreamChunk`；跟踪 block_id/block_type/tool_call_name 做关联；处理尾部 `"success"` 误报异常的吞掉逻辑。 ### 2. 统一流式协议（StreamChunk / ChunkEnum） - **`stream_chunk.py`**：`StreamChunk` 扩展为承载 AS + CC 双后端完整信息的统一结构，新增 `session_id` / `block_id` / `tool_call_id` / `tool_call_name` / `media_type` / `input_tokens` / `output_tokens` 等字段，纯文本流仍保持轻量。 - **`chunk_enum.py`**：补全生命周期标记 `REPLY_START` / `REPLY_END`，并文档化两套后端事件 → ChunkEnum 的映射。 ### 3. Index 模块重构（变化批次化 + dispatch） - 新增 `_change_batch.py`：`coalesce_changes()` 把同路径多次事件折叠为最终状态（结合 path 存在性判定），`bucket_changes()` 按 watchfiles.Change 分桶。 - 新增 `init_changes.py`（`InitChangesStep`）：一次性扫描，对比 file_store / file_catalog 已索引节点计算 added/modified/deleted，写入 `context["changes"]` 后 dispatch。 - 新增 `update_changes.py`：抽象基类 `ChangeApplyStep` 统一 added/modified/deleted 处理与错误收集；`UpdateCatalogStep`（写 file_catalog）、`UpdateIndexStep`（写 file_store，含按后缀解析 chunker）。 - **`watch_changes.py`**：改用 `dispatch_step_specs`（基类提供的 `dispatch_steps()`），每批先 `coalesce_changes` 再 dispatch；默认参数调整（debounce 5000ms / step 1000ms / poll 5000ms）并暴露常量。 - 删除旧步骤：`clear_and_scan` / `foreach_dispatch` / `scan_changes` / `update_catalog`(旧) / `update_index`(旧)；`clear_store.py` 取代 clear_and_scan。 ### 4. Evolve / Dream 模块（拆分为多步 pipeline） - 删除旧的单体 `auto_dream.py` / `dream.py` / `dream.yaml`，新增 `dream/` 子包，按 5 个步骤组织： - **`extract.py`**：扫描当日 day-index + daily 笔记，对比 file_catalog 找出 changed/deleted，调用 LLM 全局抽取 `units`（procedure/personal/wiki 三桶）与 `topics`，路径与桶做清洗/路由。 - **`integrate.py`**：逐个 unit 调用 LLM 写入 digest，结构化输出 `IntegrateOutcome`（CREATE/CORROBORATE/REFINE/CORRECT），失败 unit/路径收集回写。 - **`topics.py`**：写 `daily/<date>/interests.yaml`，结合当天已有 + 近 N 天做去重（`normalize_topic`），可走 LLM 或纯规则去重两条路径。 - **`proactive.py`**：读取当日 `interests.yaml`，作为主动推荐话题的入口。 - **`finish.py`**：把变更路径落盘到 dream file_catalog（checkpoint），渲染最终汇总摘要。 - 新增 `schema.py`（`DreamState` 等跨步骤共享状态与结构化输出模型）与 `utils.py`（状态存取、扫描打包、YAML 读写、结构化回复解析等公共函数）。 - `evolve/__init__.py` 导出全部新 step。 ### 5. auto_memory / auto_resource（适配新 Agent API） - **`auto_memory.py`**：会话路径迁移到 `<session_dir>/dialog/<session_id>.jsonl`；改用 `job_tools`；新增 `source_conversation` frontmatter 反向链接（`_session_link`）；执行后刷新 day 索引（`refresh_day_index`），并对 session_id 做合法性校验。 - **`auto_resource.py`**：资源改用「同名 daily note」方案（`_compute_note_stem` 取文件 stem）；批量处理 `changes: list[dict]`（`_handle_change` 逐项处理，返回逐项结果摘要）；agent 会话 id 用稳定的 `uuid5`；同样刷新 day 索引。 ### 6. BaseStep 基类增强 - 新增 `dispatch_steps` / `dispatch_step_specs` 机制：`_resolve_dispatch_step()` 支持字符串或 dict 形式的 step spec，`dispatch_steps()` 复用当前 context 调用下游 step。 - 新增 `config_value()`：按 key 取 app config，缺失时回退 `ApplicationConfig` 默认值。 - 小幅清理：`language` 初始化、`copy()`、`Ref.__init__` 签名精简。 ### 7. Components 改动 - **`file_store/local_file_store.py`**：持久化改用 zstd 压缩（`.jsonl.zst`，通过新 `utils/jsonl_zst.py`）；upsert 时先删除旧 chunk 的 keyword 文档；embedding 复用改为 `(text, embedding)` 键控，要求文本一致才复用；新增 `_matches_search_filter()` 对 vector/keyword 搜索做 path/path_prefix/metadata 的统一后过滤。 - **`keyword_index/bm25_index.py`**：索引文件名加入组件名 + tokenizer 指纹（sha256 前 12 位），快照/恢复时校验指纹防配置漂移；空索引 dump 时删除文件，加载失败抛错而非静默。 - **`file_chunker/markdown_file_chunker.py`**：弃用 `python-frontmatter`，改用内置 YAML 解析（非法 YAML 不阻断正文索引），并修正因 frontmatter 占用行号导致的 AST 行号偏移（`line_offset`）。 - **`cron_job.py`**：大幅简化（-187 行），由原来「dispatch 外部 job/step + 多种调度模式」改为「在自身 steps 上跑 cron 表达式」；`Application` 启动顺序随之调整为 base > stream > background > cron。 - 其余小调整：service（base/http/mcp）、file_graph、file_catalog、as_llm、as_embedding、tokenizer、prompt_handler、base_component 的签名/接口微调。 ### 8. Application 生命周期 - `_start()` 启动顺序明确为 components → base → stream → background → cron，启动失败会触发 `_close()` 回滚并 re-raise（不再吞异常）。 - 启动时创建 `session_dir` 目录；新增 `update_component()`（按类型/名就地更新已存在组件，不存在则报错）。 ### 9. File IO / 路径安全 - **`_path.py`**：`resolve_path` 增加 vault 越界防护（`is_relative_to` 校验），禁止 `.` / `..` 路径分量，支持 `allow_empty`。 - **`read.py`**：大文件（超过 `MAX_FILE_READ_BYTES`）走按行读取 `read_file_lines_safe`，避免一次性载入内存。 - **`_file_io.py` / `_daily_index.py` / `_path.py`** 等支持函数补齐（如 `refresh_day_index`、`read_file_lines_safe`）。 - **`env_utils.py`**：新增 `parse_env_file()`，`load_env()` 返回加载到的键值、支持 `override`、对无路径调用做幂等缓存。 ### 10. Config - `ApplicationConfig` 新增 `session_dir`（默认 `reme_session`）。 - `config_parser.py`：环境变量展开后做类型转换（`_convert_value`）、dot-notation 与 key=value 参数校验更严格、配置文件路径支持相对 `_CONFIG_DIR` 查找、根非 dict 报错。 - `default.yaml`：作业编排改用 `init_changes_step` + `dispatch_steps`（index/resource/digest 三个 watch loop 与 reindex）；新增 `auto_dream`（4 步）、`proactive` 作业，移除旧 `dream`；file_catalog 增配 `resource` / `digest` / `dream` 实例；LLM 默认值与 Claude Code 凭据配置调整（tool_result_limit 50000、thinking_enable=false 等）。 ### 11. 其它 - 新增 `steps/common/add.py`（`AddStep` 算术 demo）、`channel/__init__.py` 与 common `__init__` 导出整理。 - 新增 4 篇文档：`docs4/auto_dream_logic_and_step_refactor.md`、`docs4/watch_loop_step_refactor_plan.md`、`docs4/todo.md`，以及 `reme_design.md` 更新。 ** |
||
|
Sen Huang
|
f458566e2c
|
feat: add cron scheduling support and enhance Claude Code integration (#278)
* feat: add cron scheduling support and enhance Claude Code integration - Introduce CronStep for periodic job execution with support for cron expressions, daily schedules, and fixed intervals - Add automatic session management to Claude Code agent wrapper with cache-friendly defaults for system prompts and setting sources - Implement fork session support with proper validation - Enhance auto-dream functionality to dispatch per-file jobs instead of direct method calls for better backend agnosticism - Add session ID tracking to auto-resource operations - Remove deprecated download step component - Update auto-dream job naming from auto-dream to auto_dream - Add croniter dependency and update package data to include markdown files * feat: add CronJob component and rename cron step to cron job |
||
|
jinliyl
|
c3fb825af0
|
feat(agent): refactor agent wrapper, add session persistence, auto_resource step, and watch-loop improvements (#277)
* refactor(agent_wrapper): update agent wrapper implementations and config defaults
- Set default timezone to Asia/Shanghai in application config
- Add AgentScope imports and configure ReAct, context, and model configs
- Simplify __all__ export formatting in agent wrapper init
- Remove redundant docstring details from agent wrapper classes
- Optimize tool result handling with state assignment simplification
- Add permission context and state management for AgentScope backend
- Update Claude Code wrapper tool creation and server registration logic
- Configure default agent settings including permission mode and retry limits
- Remove obsolete comments and streamline code structure
* fix(agent): add output schema validation and BaseModel support
- Added type assertion to ensure output_schema is a dict in as_agent_wrapper
- Imported BaseModel from pydantic in base_agent_wrapper
- Modified set_output_schema to accept both dict and BaseModel types
- Added automatic conversion of BaseModel to JSON schema
- Updated method documentation to reflect new type support
* refactor(agent): replace direct agent instantiation with agent wrapper component
- Removed manual Agent creation and initialization in llm_demo step
- Integrated agent_wrapper component as dependency in base step
- Updated llm_demo step to use agent_wrapper.reply method instead of direct agent calls
- Modified structured output handling to work with new agent wrapper interface
- Simplified agent configuration by using wrapper's built-in functionality
- Updated documentation to reflect agent wrapper usage instead of direct as_llm access
- Removed redundant imports related to manual agent management
* feat(agent): add streaming support and refactor agent wrapper components
- Introduce reply_stream method in base agent wrapper with fallback implementation
- Add _build_agent helper method to AsAgentWrapper for agent instantiation
- Implement structured output generation with proper model assertions
- Update StreamLLMDemoStep to use agent_wrapper instead of direct Agent calls
- Replace manual streaming logic with execute_stream_task utility function
- Change default system prompt to provide detailed responses instead of concise ones
- Add colored output support for different chunk types in streaming demos
- Refactor test cases to use async task execution with streaming verification
* refactor(agent): remove session_id parameter from reply methods
- Removed session_id parameter from ASAgentWrapper.reply method signature
- Removed session_id parameter from BaseAgentWrapper.reply abstract method
- Removed session_id parameter from CCAgentWrapper.reply method signature
- Updated reply_stream methods to remove session_id parameter across all wrappers
- Modified CCAgentWrapper to use dynamic options assignment instead of hardcoded properties
- Set default system_prompt in config instead of hardcoded in code
- Increased default max_turns from 10 to 50 in configuration
* config: update default configuration and script entry point
- Change resource_dir from empty string to 'resource'
- Update command line entry point from 'reme4' to 'reme'
* feat(agent): add session state persistence and forking support
- Implement AsStateHandler for AgentState JSONL serialization
- Add session_id parameter to AsAgentWrapper.reply method
- Create timestamp-based session file paths with timezone support
- Load existing session state from JSONL files when session_id provided
- Save updated session state after each agent interaction
- Support session forking with UUID generation for new sessions
- Add integration tests for session persistence and forking scenarios
- Include temporary directory utilities for testing isolated sessions
- Ensure parent directories are created for session files automatically
* refactor(auto_memory): replace transcript parsing with direct message handling
- Remove transcript loading logic and related dependencies
- Add session message saving functionality with deduplication
- Use agent wrapper instead of direct AgentScope agent instantiation
- Simplify timezone handling using shared now utility
- Update logging and response metadata structure
- Remove unused imports and toolkit management methods
- Change session file naming from session_{id}.jsonl to session_agent_{id}.jsonl
* refactor(steps): move channel steps from index to channel module
- Move ChannelNotifyStep from .index.channel_notify to .channel.channel_notify
- Move ClaimChannelStep from .index.claim_channel to .channel.claim_channel
- Update __init__.py imports to reflect new module structure
- Reorganize steps list in __init__.py with channel section before index
- Add proper file prefix handling in daily index processing
- Update test imports to use new channel module location
* feat(evolve): add auto_resource step for interpreting resource files
- Add AutoResourceStep to interpret resource files into daily notes via an agent
- Implement resource file parsing with date and filename extraction logic
- Add session ID computation using MD5 hash of filename
- Create delete and upsert handlers for resource file operations
- Add truncation and sanitization functions for tool output in auto_memory
- Register auto_resource step with proper parameter validation
- Add configuration for resource watch loop with file extension filters
- Update default YAML config to include resource watch and digest watch loops
- Add shared watch-rule logic for scan_changes and watch_changes steps
- Implement foreach_dispatch and log_changes steps for change processing
- Rename update_store_index_loop to index_update_loop in configuration
- Refactor file chunking interface from parse to chunk method
- Remove unused imports and dependencies in auto_dream step
- Fix path iteration formatting in daily_index utility function
- Add comprehensive integration tests for auto_resource functionality
* refactor(auto_resource): format function call with multi-line parameters
- Reformatted await _handle_upsert call to use multiple lines for better readability
- Removed unused imports from scan_changes.py including BaseFileCatalog and ComponentEnum
- Added date parameter to RuntimeContext initialization in test cases
- Updated expected file paths in test assertions to include session_agent prefix
- Formatted long assertion statements across multiple lines to maintain character limit
- Corrected wikilink references from generic names to session_agent prefixed names
|
||
|
jinliyl
|
8eaa96390a
|
refactor(file_chunker): replace file parser with file chunker component (#276)
* refactor(file_chunker): replace file parser with file chunker component - Rename file_parser module to file_chunker across codebase - Update BaseFileParser to BaseFileChunker with corresponding component type - Rename LinkedFileParser to MarkdownFileChunker for markdown-specific chunking - Rename ChunkedFileParser to DefaultFileChunker for default byte-based chunking - Update documentation references from file_parser to file_chunker - Modify dependency injection in BaseStep to use file_chunker instead of file_parser - Update configuration and component registration to use new chunker naming - Rename all related test files and update test assertions accordingly - Add recursive option to scan_store_changes_step in default configuration * feat(database): enhance Neo4j connection with environment variable support - Add support for NEO4J_PASSWORD environment variable as fallback - Make password parameter optional in constructor with validation - Update chromadb dependency from 1.3.5 to 1.5.7 - Configure CORS credentials based on origin settings - Import os module for environment variable access * feat(config): add timezone support and remove unused dialog directory - Added timezone field to application config with IANA timezone support - Removed unused dialog_dir configuration and related directory creation - Replaced date.today() with timezone-aware now() function across daily operations - Created evolve module with timezone-aware datetime functionality - Updated daily_create, daily_list, and daily_reindex steps to use timezone-aware dates * refactor(steps): update file chunker implementation - Replace ChunkedFileParser with DefaultFileChunker in background steps - Add module docstring to evolve steps package - Update return type annotation to reflect new chunker class usage * refactor(components): rename embedding and llm components to as_embedding and as_llm - Rename reme4/components/embedding to reme4/components/as_embedding - Rename reme4/components/llm to reme4/components/as_llm - Update all imports and references from embedding to as_embedding - Update all imports and references from llm to as_llm - Change BaseEmbedding to BaseAsEmbedding and update inheritance - Change BaseLLM to BaseAsLLM and update inheritance - Update component types from LLM/EMBEDDING to AS_LLM/AS_EMBEDDING - Update configuration keys from embedding/llm to as_embedding/as_llm - Update all property references from llm to as_llm in step classes - Update test assertions to use new component enum values * refactor(embedding_store): rename embedding parameter to as_embedding - Updated configuration key from 'embedding' to 'as_embedding' - Renamed class attribute from 'embedding' to 'as_embedding' - Updated method calls to use 'as_embedding' instead of 'embedding' - Changed parameter name in constructor from 'embedding' to 'as_embedding' - Updated documentation to reflect new parameter name - Modified health check to use 'as_embedding' property * feat(agent_wrapper): add unified agent wrapper component with multiple backends - Introduce BaseAgentWrapper abstract base class for agent implementations - Add AsAgentWrapper implementation using AgentScope framework - Add CcAgentWrapper implementation using Claude Code SDK - Register agent_wrapper component type in ComponentEnum - Configure default agent_wrapper settings in default.yaml - Implement tool integration for both AgentScope and Claude Code backends - Support fluent configuration via set_system_prompt() and add_tools() methods * feat(agent-wrapper): add structured output support for agent wrappers - Import SystemMsg in AsAgentWrapper for structured output handling - Add output_schema parameter support in AsAgentWrapper with generate_structured_output - Implement set_output_schema method in BaseAgentWrapper for chaining configuration - Add output schema support in CcAgentWrapper with JSON schema format option - Return structured output when available in CcAgentWrapper response - Refactor kwargs handling to use default values consistently across wrapper classes |