* feat(service): expose MCP through HTTP backend
Serve JSON/SSE job endpoints and streamable HTTP MCP from one FastAPI application, sharing the same jobs and application lifecycle. Preserve the standalone MCP backend, add configurable MCP HTTP settings, update startup metadata and integration docs, and cover routing, lifecycle, configuration, and compatibility behavior with unit tests.
* fix(service): preserve MCP request protections
Route the exact MCP path through the complete FastMCP ASGI application so its middleware and state remain active. Reject non-literal MCP paths and validate reserved Job conflicts before tolerant service registration. Add regression coverage for middleware preservation, route syntax, and startup failure.
* fix(service): reject encoded MCP paths
Reject percent signs in mcp_path so ASGI path decoding cannot turn an accepted configuration into an unreachable route. Cover encoded slash, space, and double-encoded slash inputs.
* feat(backend): improve workspace support for web clients
* fix(config): preserve the default workspace directory
* chore(reme): bump version to 0.4.1.5
- Update __version__ from 0.4.1.4 to 0.4.1.5 in initialization file
* fix(chat): disable builtin tools in read-only mode
* fix(agent): make builtin tools opt-in
* fix(list): tolerate files removed during mtime sort
* fix(chat): expose complete read-only job set
* feat(file_store): upgrade FAISS to HNSW index with async reindex and path constraint
- Replace IndexFlatIP with IndexHNSWFlat for better recall/speed tradeoff
- Add dynamic efSearch (limit * 5) scaled to query request size
- Add async_reindex option: background rebuild with generation-based invalidation
- Extract _delete_nodes() in LocalFileStore for subclass reuse
- Add unit tests for file store consistency
* fix: resolve pylint warnings in faiss store and test file
* refactor(file_store): replace generation-based reindex with event-flag worker
- Replace _reindex_generation/lock/task with a single long-lived worker
coroutine consuming an asyncio.Event flag; repeated submissions coalesce
- Use local index reference in vector_search to avoid TOCTOU on self._faiss_index
- Pass index explicitly to _set_ef_search for consistency
- Track _index_writes to re-arm reindex after concurrent writes
- Update tests to match new internal API
* fix: resolve pylint too-many-return-statements and implicit-booleaness warnings
* feat(file_store): add refine maintenance hook and incremental embedding backfill
- Add refine() idle-time maintenance hook to BaseFileStore/LocalFileStore
- FaissLocalFileStore: incremental vector add on backfill instead of full rebuild
- Dynamic tombstone compaction threshold scaled by index size
- Add RefineStoreStep with daily cron job (refine_store_cron)
- Enable faiss backend and embedding_store by default in default.yaml
- Add unit tests for faiss index maintenance
* chore(deps): promote faiss-cpu to core dependencies
faiss backend is now the default file_store, so faiss-cpu moves from
the optional [core] extra to the base dependencies list.
* feat: rename refine_store to optimize_index and add vecdb_path_constraint
- Rename refine_store step to optimize_index with cron job scheduling
- Add vecdb_path_constraint to file_store components
- Update default.yaml with optimize_index_cron and faiss backend comment
- Update memory_search docs (en/zh) for FAISS vector management
- Update unit tests for index maintenance
* feat(faiss): add embedding digest to reject stale sidecar after partial dump
Add _chunks_embedding_digest() that computes an order-independent SHA-256
over (chunk_id, float16 embedding) pairs. The digest is written into the
idmap sidecar at dump time and verified at load time. A mismatch means the
sidecar vectors belong to a different chunk generation than the authoritative
JSONL — detectable even when the live-ID set is unchanged (same-ID in-place
update crash window).
Add test_faiss_rejects_stale_sidecar_after_partial_dump reproducing the
crash-between-writes scenario and asserting digest-based rejection.
Compress verbose docstrings/comments in existing tests for pylint line
budget.
---------
Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
* feat(config): add environment variable configuration for agent subprocesses
- Add environment field to ApplicationConfig to store variables for agent subprocesses
- Remove dynamic loading of .env files in agent wrappers
- Introduce subprocess_environment property in base agent wrapper
- Pass application-level environment variables to Claude Code and Codex agents
- Load environment variables once at startup and pass to ReMe application
- Remove dependency on load_env utility in agent wrapper implementations
- Update tests to use configured environment instead of dynamic loading
- Remove unused environment loading utilities and related test cases
* refactor(mcp): remove channel notification system and related components
- Removed channel notification step implementation
- Removed claim channel step implementation
- Removed ChannelSink class from MCP service
- Removed channel-related documentation from AGENTS.md
- Removed channel instruction text from MCP service
- Removed all channel-related tests
- Updated application context metadata comment to remove channel sink reference
- Removed channel module initialization and imports
* feat(service): add job whitelisting capability to BaseService
- Add optional jobs parameter to BaseService.__init__ to configure job whitelist
- Store jobs as set in self.jobs attribute for efficient lookup operations
- Modify add_jobs method to filter jobs based on whitelist configuration
- Update documentation in both English and Chinese to describe new feature
- Add comprehensive unit tests for job whitelisting behavior
- Implement flowchart update showing new filtering logic
- Preserve existing enable_serve flag behavior alongside new whitelisting
* refactor(service): enhance service job validation and MCP tool injection
- Add strict validation for service jobs whitelist with detailed error messages
- Implement injected job arguments support for MCP services with conflict detection
- Add tool error handling for unsuccessful responses in MCP services
- Remove duplicate job names in Codex agent wrapper using dict.fromkeys
- Update MCP server argument format from single JSON array to repeated --job flags
- Add comprehensive test coverage for job injection and error handling scenarios
- Update documentation to reflect service job validation and MCP features
- Ensure application cleanup occurs even when service lifespan encounters errors
* feat(agent): update skill handling to preserve existing Claude skills
- Change skills parameter processing to use 'all' instead of filtered list
- Add logic to select project skills without restricting Claude's existing skills
- Update variable naming from 'skills' to 'selected_skills' for clarity
- Modify application context metadata documentation to clarify in-memory state usage
- Add test case to verify configured skills are added without filtering existing skills
- Update internal skill directory handling to use renamed variable consistently
* refactor(agent): restructure agent wrapper components and session storage
- Move CcFileSessionStore to separate module for better organization
- Add SDK package version logging in base agent wrapper
- Update Claude Code agent to use new session store structure with project keys
- Refactor Claude Code agent wrapper to use proper type hints and SDK integration
- Add support for server tool use events in Claude Code message processing
- Improve error handling and resource cleanup in streaming operations
- Update Codex agent wrapper with proper type annotations and configuration
- Remove deprecated system prompt mode handling from Claude Code wrapper
- Fix session path construction for Claude Code transcript storage
- Update dependency injection and configuration handling patterns
* fix(cc_agent_wrapper): resolve Claude Code SDK integration issues
- Added dataclass import and created _BlockState for content block metadata tracking
- Implemented proper MCP server name constant and tool context ID validation
- Fixed tool_context_id injection to prevent duplicate assignment errors
- Resolved skills parameter handling in build_options method
- Enhanced job tools integration with MCP servers mapping validation
- Replaced deprecated block_ids/block_types/tool_call_names with block_states dict
- Updated message_delta to emit USAGE chunks instead of REPLY_END
- Fixed stream result handling to ensure proper REPLY_END emission
- Improved error handling for session mirror failures and rate limits
- Added proper cleanup for expected trailing errors in streams
- Refactored Codex agent wrapper initialization and configuration management
- Removed obsolete system_prompt_mode from default config
- Enhanced test coverage for new block state and error handling features
- Fixed async generator handling with aclosing context manager
- Improved chunk type mapping for Claude Code SDK events
* refactor(tests): remove demo config tests from config parser test suite
- Removed test_demo_config_registers_llm_jobs function and its assertions
- Eliminated verification of LLM demo job configurations
- Removed checks for agent wrapper component settings
- Deleted assertions for model configurations and parameters
- Cleaned up deprecated test cases related to demo config parsing
* refactor(evolve): simplify Claude Code session store path structure
- Removed redundant project key subdirectory from session link generation
- Updated CcFileSessionStore initialization to use direct session directory path
- Maintained existing session layout compatibility for backward compatibility
- Added unit tests to verify session persistence behavior with existing transcripts
- Ensured UUID-based session files remain accessible at expected locations
- Preserved existing session directory structure without additional nesting
* refactor(agent): defer optional Codex SDK imports until first use
- Moved openai-codex imports inside functions to avoid mandatory dependencies
- Added TYPE_CHECKING guard for development time type checking only
- Implemented lazy loading mechanism with _get_async_codex_class function
- Updated AsyncCodex initialization to occur on demand rather than at module level
- Maintained backward compatibility while improving import performance
- Added test case to verify package import works without optional Codex SDK
- Updated agentscope dependency to version 2.0.4.post1 in pyproject.toml
* test(embedded): add compatibility tests for in-process ReMe embedding
- Add test suite for QwenPaw-style embedded configurations
- Verify optional defaults remain preserved in embedded configs
- Ensure in-process application API stays compatible
- Test model injection and lifecycle management compatibility
- Remove obsolete hermes agent plugin tests
- Update CLI import test to cover multiple optional SDKs
- Block claude_agent_sdk and openai_codex during import testing
* feat(codex): add authentication mode support with thread-safe logging
- Implement _CodexAuthConfig dataclass for resolved auth settings
- Add auth_mode parameter with auto/api_key/oauth options
- Separate API key and OAuth authentication flows
- Force specific login method based on auth mode
- Add explicit API key validation requirement
- Serialize concurrent logger initialization in thread lock
- Close logging handlers properly during cleanup
- Update default config with auth_mode presets for codex and codex_oauth
- Add comprehensive tests for authentication modes and concurrent logging
* feat(file_store): implement background embedding backfill and keyword index repair
- Add _after_embedding_backfill hook in FAISS local file store
- Schedule startup embedding repair without delaying component readiness
- Cancel and collect embedding backfill task during component shutdown
- Log progress at fixed percentage boundaries for long-running operations
- Process embedding backfill in configurable batch sizes with progress reporting
- Rebuild keyword index in bounded batches with detailed mismatch diagnostics
- Format stdlib logs consistently with QwenPaw console output using relative paths
- Run embedding backfill as background task that doesn't block component startup
- Add comprehensive tests for background embedding and keyword index repair scenarios
* fix(file-store): repair graph-chunk consistency on load
- Add _repair_graph_chunk_consistency method to detect and fix mismatched graph/chunk states
- Clear torn graph/chunk state when missing or orphaned chunks are detected
- Ensure keyword index sync handles empty chunks properly
- Add comprehensive tests for graph-chunk consistency scenarios
- Update test utilities to properly seed graph/chunk snapshots
- Increment version to 0.4.1.2
* feat(file_io): enhance list step response format and add comprehensive logging
- Format list output with bullet points for better readability
- Add explicit "No files found" message when directory is empty
- Include detailed timing information for file store startup phases
- Add logging for chunk loading, graph consistency checks, and keyword indexing
- Provide detailed metrics for embedding backfill operations
- Add comprehensive test coverage for empty directory scenarios
- Include batch processing statistics for embedding operations
* feat(logger): add QwenPaw logging integration with forwarding mechanism
- Introduce _ForwardToLoggerHandler to forward log records to target logger
- Add qwenpaw logger integration that forwards ReMe logs to QwenPaw handlers
- Maintain ReMe logger stability for modules that cache it at import time
- Enable QwenPaw handlers to take effect without ReMe reconfiguration
- Add comprehensive tests for stdlib forwarding to QwenPaw sinks
- Support explicit REME_DISABLE_LOGURU=false to keep original Loguru backend
- Preserve existing logging behavior when QwenPaw is not configured
* fix(file_store): serialize concurrent FAISS dump operations to prevent corruption
- Add asyncio lock to ensure only one FAISS dump operation runs at a time
- Generate unique temporary filenames using UUID tokens for atomic replacement
- Implement proper cleanup of temporary files in finally block
- Add comprehensive test to verify concurrent dumps are serialized
- Ensure atomic writes by replacing both index and idmap files together
- Prevent partial state writes during concurrent access scenarios
- Removed max_file_bytes configuration from index_update_loop, resource_watch_loop, digest_watch_loop, and reindex jobs
- Updated default.yaml to reflect simplified job configurations without file size limits
- Removed corresponding test case that validated the 20 MiB limit behavior
- Simplified watch directories and suffixes to basic configurations
- Cleaned up unnecessary commented configurations in the YAML file
* feat(benchmark): add final answer review step for evaluation
- Introduce FinalAnswerReviewStep to handle answer validation
- Add final_answer_review.jsonl dataset with 24 evaluation cases
- Include detailed reasoning and golden check results for each case
- Support various question types including temporal reasoning and preferences
- Implement time consistency checks for session references
- Add comprehensive test coverage for different evaluation scenarios
* chore(benchmark): remove longmemeval final answer review file
- Removed final_answer_review.jsonl containing 23 evaluation records
- Deleted question_id mappings with detailed reasoning for golden answers
- Removed answer correctness assessments and session time validation checks
- Cleaned up benchmark dataset used for memory evaluation testing
- Eliminated JSONL format evaluation results for temporal reasoning tasks
- Removed references to various session IDs and time-based validations
* config(default): disable shell step configuration by commenting out
- Commented out the shell step configuration in default.yaml
- Disabled asynchronous shell command execution capability
- Removed shell step from available backend operations
- Preserved traverse backend configuration unchanged
* refactor(tests): remove unused shell job test from config parser tests
- Removed test_default_config_registers_shell_job function that was no longer needed
- Kept existing test for frontmatter chunk metadata configuration
- Cleaned up test suite by removing obsolete test case
* feat(index): add file size limits and oversized file handling
- Implement max_file_bytes configuration option for content processing jobs
- Add default 20MB file size limit for background processing in default config
- Skip oversized files during auto_resource step with appropriate metadata
- Clear stale index entries when oversized files are modified
- Add size-based filtering logic to update_changes step with skip reporting
- Include file size validation in UpdateIndexStep with proper response handling
- Add comprehensive tests for oversized file scenarios in auto_resource and update_index
- Document file size limits in constants with appropriate thresholds
* chore(version): bump version to 0.4.1.1
- Update __version__ from 0.4.1.0 to 0.4.1.1 in __init__.py
* fix(index): isolate batch metadata and handle file races
* feat(core): add shell command execution and memory status reporting
- Introduce ShellStep for executing shell commands with timeout support
- Add StatusStep to report memory estimates for stateful data components
- Register shell and status commands in default configuration
- Update documentation with new reme status and shell command capabilities
- Implement comprehensive unit tests for both new step types
- Add support for asynchronous command execution with proper error handling
* feat(config): add log_config option to suppress config loading logs
- Add log_config parameter to resolve_app_config function with default True
- Conditionally log config loading messages based on log_config flag
- Update reme.py and service_utils.py to use log_config=False for client calls
- Suppress config logging in user-facing contexts to avoid output pollution
refactor(shell): rename command parameter to cmd for clarity
- Change 'command' to 'cmd' in default.yaml configuration schema
- Rename 'timeout' to 'shell_timeout' to avoid parameter name collisions
- Update ShellStep to accept both legacy and new parameter names
- Maintain backward compatibility with existing command/timeout usage
test(shell): add comprehensive tests for shell step parameter handling
- Add test cases for new cmd and shell_timeout parameter names
- Verify legacy command and timeout parameters still work
- Test blank command rejection message updated to use cmd
- Create integration test for shell parameter payload passing
* fix(shell): ensure proper environment loading and process timeout handling
- Move load_env() call to execute before parse_args() in main function
- Add proper process group killing for timeout scenarios on POSIX systems
- Implement recursive child process termination on Windows for proper cleanup
- Change parameter name from 'timeout' to 'shell_timeout' in shell execution
- Remove support for legacy 'command' and 'timeout' parameter names
- Update test cases to verify new timeout behavior and parameter requirements
- Add comments explaining component size tracking implementation details
- Set default version to 0.4.1.0
- Comment out embedding configuration in default.yaml
- Update README and README_ZH to clarify embedding components are disabled by default
- Add note explaining how to enable embedding-based semantic retrieval
- Adjust table formatting and descriptions in documentation
- Modify search command description to reflect vector search availability when enabled
* refactor(embedding): update embedding model initialization and session storage paths
- Remove unused inspect import from as_embedding module
- Pass dimensions directly to embedding model constructor instead of using parameters
- Update session state file paths to use mem_session directory instead of resource
- Add mem_session_dir configuration option to application config schema
- Update workspace directory creation to include new mem_session directory
- Change AgentScope and Claude Code session paths to use mem_session directory
- Move embedding dimensions from parameters to top-level configuration
- Update AgentScope dependency version from 2.0.3 to 2.0.4
- Update integration tests to reflect new session file location paths
* chore(version): bump version to 0.4.0.8
- Update __version__ from 0.4.0.7 to 0.4.0.8 in __init__.py
* feat(search): add tool context deduplication and improve search configuration
- Modify _make_tool methods to accept and inject tool_context_id parameter
- Add tool_context_id handling in AS and CC agent wrappers
- Increase search candidate multiplier from 3.0 to 5.0 in default config
- Extend HTTP client timeout from 30s to 3600s
- Add tool context deduplication logic to prevent duplicate search results
- Implement TTL-based expiration for seen chunks in tool contexts
- Add comprehensive unit tests for tool context deduplication behavior
- Update .gitignore to exclude longmemeval directory
- Add time import for timestamp functionality in search step
* refactor(search): replace time module with datetime for timestamp generation
- Removed unused time import
- Added static method _now_ts using datetime.timestamp
- Updated clock parameter to use _now_ts method instead of time.time
- Maintained same timestamp precision and functionality
* feat: add start_date/end_date time filter support for search job
- Add _extract_date_from_path to extract validated YYYY-MM-DD from chunk paths
- Add start_date/end_date filtering in _matches_search_filter
- Implement progressive recall in FaissLocalFileStore.vector_search
- Promote start_date/end_date from context to search_filter in SearchStep
- Add start_date/end_date parameters to search job in default.yaml
- Add unit tests for date filter functionality
* fix: validate/normalize date filters and harden _extract_date_from_path
Address three code-review comments on the time_filter search feature:
1. Validate/normalize start_date and end_date before string comparison.
_matches_search_filter does lexicographic comparison against path_date
(always canonical YYYY-MM-DD). Raw caller values like '2026-2-28' or
'abc' would produce silently wrong results. Now SearchStep normalizes
valid dates via extract_daily_date (with strptime fallback for
non-zero-padded input) and silently ignores invalid dates with a
logger.warning, removing them from the filter.
2. Clarify behavior for paths without embedded dates.
Added optional strict_date_filter parameter (default False). When True
and at least one date bound is active, chunks whose path yields no date
(e.g. digest/personal/topic.md) are excluded. When False (default),
the existing behavior is preserved — dateless paths pass through.
3. Harden _extract_date_from_path against non-standard suffixes.
Previously parts[1].split('.')[0] accepted '2026-05-18.anything' as a
valid date. Now only exact 'YYYY-MM-DD' (dir) and 'YYYY-MM-DD.md'
(day-index) forms are accepted.
* fix(auto-memory): preserve message timestamps
* fix(auto-memory): infer daily date from messages
* feat(file_io): add strict date parsing and improve daily date handling
- Add new parse_daily_date function for strict YYYY-MM-DD validation
- Replace extract_daily_date with parse_daily_date for explicit date validation
- Change _messages_day to use max date instead of min for historical imports
- Reorder imports to maintain consistent module ordering
- Move session message saving after date validation in auto_memory
- Add comprehensive tests for invalid date rejection before saving
- Add tests for strict YYYY-MM-DD date format validation
- Update test names to reflect latest date behavior
---------
Co-authored-by: Ziyang Guo <121015044+RunMarshal@users.noreply.github.com>
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
* feat(file_io): add daily_write step for creating daily notes with conversation metadata
- Add DailyWriteStep class that delegates to write job for creating daily notes
- Register daily_write job in default configuration with proper parameters
- Include validation for name and session_id path components
- Add test coverage for daily_write functionality including metadata handling
- Preserve existing job execution method in application.py after repositioning
- Update base_step.py to use positional-only parameter syntax for job methods
- Import and expose DailyWriteStep in file_io module initialization
- Override reserved metadata keys (name, description, session_id, source_conversation) with fixed values
- Refresh daily index after successful write operation
- Generate proper source conversation links in markdown format
* feat(daily): refactor daily note system with enhanced metadata handling
- Introduce validate_filename_component function and export it
- Add _INDEX_HIDDEN_METADATA_KEYS to hide conversation metadata from index
- Update scan_notes to exclude hidden metadata keys from index rendering
- Modify auto_memory to use daily_write tool and manage session frontmatter
- Implement session note lookup and renaming based on frontmatter name
- Update daily_list to return flattened note metadata including session info
- Change daily_write to dispatch write step instead of running job
- Add test cases for updated daily note functionality and metadata handling
- Update version from 0.4.0.2 to 0.4.0.3
* fix(evolve): correct metadata update in auto memory response
- Fixed trailing comma issue in metadata dictionary update
- Ensured proper formatting of response metadata structure
- Maintained existing functionality while fixing syntax error
* refactor(auto_resource): replace daily_create with dynamic note management
- Remove DailyCreateStep and related exports from file_io module
- Replace static daily note creation with dynamic resource-linked card system
- Implement LLM-suggested naming with frontmatter-driven file management
- Add source_resource linking for tracking original files
- Introduce collision handling with hash-based suffixes
- Update documentation to reflect new resource card workflow
- Modify auto_resource prompts to use write/edit tools instead of daily_create
- Adjust test fixture comments to match new agent behavior
- Update framework diagrams and quick start examples accordingly
* feat(app): add version info to app initialization and update auto-memory logic
- Include version number in application startup logging
- Remove tool result truncation logic from auto-memory step
- Update auto-memory to exclude tool_result blocks from saved history
- Add test case to verify tool results are filtered out from message saving
- Update YAML prompts to clarify filename naming rules without dates
- Modify configuration to support new dispatch steps format with persistence control
* feat(auto_memory): add note modification tracking and optimize frontmatter updates
- Add _note_bytes and _note_modified methods to track actual file changes
- Optimize frontmatter updates by checking existing metadata before update
- Add modified flag to response metadata indicating actual note changes
- Update logging to include modified status in various operations
- Add comprehensive tests for modified/unmodified detection scenarios
- Enhance result hook logic to skip when no actual changes occur
- Refactor metadata handling to properly track creation vs modification status
* feat(evolve): enhance agent reply processing and logging capabilities
- Add agent_reply_result_text function to extract final user-visible text from agent replies
- Implement comprehensive logging throughout auto_memory, auto_resource, and dream modules
- Add max_units configuration option to limit extracted memory units
- Improve error handling and validation in auto_resource step
- Refactor dream extract step to respect max_units limit during processing
- Enhance summary rendering in dream finish step with detailed breakdown
- Add result hook functionality for embedding hosts integration
- Implement loose resource filename handling for root-level resources
- Update test cases to reflect new functionality and improved error messages
* test(background-steps): update fake upsert function to include created parameter
- Modified fake_upsert function to accept 'created' parameter instead of '_created'
- Added 'created' field to captured dictionary in fake_upsert function
- Included 'created': True in the expected response dictionary for test case
- Updated test assertion to match new parameter structure
* fix(core): update version number to 0.4.0.1
- Incremented version from 0.4.0.0 to 0.4.0.1 in __init__.py
* fix(index): remove stopwords path from tokenizer config and add keyword index repair
- Remove stopwords_path from tokenizer config to prevent index forking by install path
- Add _sync_keyword_index_from_chunks method to repair keyword index when persisted state mismatches
- Implement test for keyword index repair from persisted chunks when missing
- Add test to verify tokenizer fingerprint ignores stopwords absolute path
- Update version from 0.4.0.1 to 0.4.0.2
* feat(dream): add scan_days parameter to dream extraction process
- Add scan_days configuration option to default.yaml with default value of 2
- Implement recent_dates utility function to calculate date ranges for scanning
- Modify DreamExtractStep to scan multiple days based on scan_days parameter
- Update dream extraction to process files across multiple dates instead of single day
- Extend DreamState schema to include dates and scan_days fields
- Update DreamTopicsStep to handle multi-day topic processing
- Modify finish step to checkpoint files from all scanned dates
- Add comprehensive tests for multi-day scanning functionality
- Update prompt templates to include scan dates information
- Refactor topics writing logic to target specific date rather than current date
* docs: rename vault_dir to workspace_dir in documentation and examples
* refactor(extract): format long method call across multiple lines
* refactor(extract): format system prompt parameters for better readability
