* feat(daily-paper): add Hugging Face mirror switch
* refactor(daily-paper): simplify the HF mirror switch and warn on ignored env
The switch was a three-state bool|None where None preserved the legacy
environment-driven selection, but no production caller ever passes None --
collect.py always resolves an explicit bool. Collapse it to a plain bool
defaulting to False.
HF_MIRROR_URL no longer redirects traffic on its own, so warn when it is
configured while the mirror stays disabled; a mirror-only setup would
otherwise fall back to the official site with no signal. Both READMEs now
record the behavior change and stop presenting the two mirror variables as
symmetric -- arXiv remains environment-driven while Hugging Face is gated on
the job parameter.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(daily-paper): address mirror configuration feedback
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* feat(file_store): add ZvecLocalFileStore backend
- Implement ZvecLocalFileStore with native zvec collection for ANN search.
- Keep JSONL chunks as the source of truth; rebuild collection from chunks
when sidecar digest/dimension/HNSW M mismatch is detected.
- Add dedicated unit tests in tests/unit/test_zvec_file_store.py.
- Parametrize existing file_store consistency tests to cover both
LocalFileStore and ZvecLocalFileStore.
- Register the new backend in reme/components/file_store/__init__.py.
* fix(file_store): fix zvec collection sync and content validation, declare zvec dependency
* feat: add ssh proxy
* feat: add ssh proxy
* feat: add ssh proxy
* feat: add ssh proxy
* feat: add prompt
* feat: add agent wrapper
* feat: add agent wrapper
* feat: add agent wrapper
* feat: add tushare skill
* feat: add tushare skill
* feat: add tushare skill
* feat: add none stream
* chore(deps): update dependency versions in pyproject.toml
- Bump claude-agent-sdk from 0.2.123 to 0.2.126
- Upgrade pre-commit to version 4.6.1 or higher
- Upgrade pytest to version 9.1.1 or higher
* feat(agent_wrapper): add session compaction support and unify session commands
- Introduce compact_session method to BaseAgentWrapper and implement it in AsAgentWrapper, CcAgentWrapper, and CodexAgentWrapper
- Add session_command module with SessionCommandResult dataclass and handle_session_command function for /clear and /compact commands
- Update __init__.py exports to include session_command handlers
- Modify DingTalkWaitStep to handle session commands via handle_session_command function
- Remove streaming mode from DingTalkWaitStep and simplify reply handling to final Markdown replies only
- Add unit tests for session compaction methods and session command handling across wrappers and DingTalk integration
- Clean up and remove obsolete streaming and card rendering code from DingTalk wait step
- Adjust daily_cookbook.yaml to remove stream and card_update_interval config entries for DingTalk wait step
* feat(auto_fin): add Auto Fin simulated portfolio cookbook workflow
- Add comprehensive Auto Fin schema exports for multiple models and enums
- Implement base class and helpers for Auto Fin analysis steps
- Create file, state, and formatting utilities for Auto Fin with atomic file writes and locking
- Define Auto Fin pipeline with four analysis agents: backtest, event, portfolio, and US correlation
- Register Auto Fin package in cookbook workflows and schema initialization
- Add detailed documentation in markdown describing the system design, workflow, and data contracts
* feat(outbound_proxy): add application-scoped outbound HTTP proxy components
- Introduce BaseOutboundProxy and OutboundProxyEndpoint as core contracts
- Implement FixedHttpOutboundProxy for external HTTP proxy integration
- Add SshHttpOutboundProxy providing SSH-backed local HTTP proxy tunnels
- Register outbound proxy components in component registry and enumeration
- Update components package to include outbound_proxy module
- Add dependency on pproxy for SSH HTTP proxy bridging
- Include comprehensive unit tests covering proxy lifecycle, validation,
environment merging, error handling, readiness, and monitoring mechanisms
* refactor(network): replace SSH proxy with explicit HTTP outbound proxy
- Remove SSH proxy helper implementation and references in codebase
- Add support for explicit HTTP proxy URL in arXiv and HuggingFace clients
- Modify clients to use async context manager for consistent resource handling
- Update daily paper steps to forward outbound proxy configuration explicitly
- Change tests to cover new proxy usage model and remove SSH proxy mocks
- Add outbound proxy component configuration in daily_cookbook.yaml
- Ensure proxy URL usage disables environment trust in HTTP clients
- Fix app context component enum access to be defensive against missing keys
* feat(agent_wrapper): add managed proxy support for command environments
- Introduce BaseOutboundProxy binding in BaseAgentWrapper for outbound proxy management
- Add bash_environment and command_proxy_environment properties to apply proxy settings
- Update WorkspaceBackend instantiation in AsAgentWrapper to use bash_environment
- Inject managed proxy export commands into Claude Code Bash commands via hooks
- Enhance CodexAgentWrapper to include managed proxy in shell environment policy
- Modify daily_cookbook.yaml steps to specify outbound_proxy as default where needed
- Add comprehensive unit tests verifying managed proxy injection and environment isolation
- Ensure subprocess_environment remains unchanged while proxy is applied selectively to commands
* refactor(memory): replace search job_tools with memory in daily cookbook config
- Change workspace_dir default from .reme to reme_workspace
- Replace search job_tools with memory across multiple components and jobs
- Update descriptions to reflect long-term memory retrieval instead of search
- Modify system prompts to instruct using memory for retrieving notes
- Adjust unit tests to verify memory job_tools and job presence instead of search
- Ensure consistency in configuration and tests for memory backend usage
* refactor(config): rename memory to memory_search in daily cookbook config
- Change all occurrences of "memory" to "memory_search" in job_tools and job definitions
- Update related system prompts to reflect the new memory_search terminology
- Modify unit tests to assert the presence of memory_search instead of memory
- Ensure consistency across skills, job tools, and backend configurations in multiple components
* feat(auto_fin): add deterministic quantitative research and ranking fusion
- Introduce new schema models: EtfScore, RankingMetrics, ExtremeAnalysis,
DimensionRanking, and FusionRanking to represent deterministic research outputs
- Add ranking data to event, backtest, us_correlation, and portfolio analysis outputs
- Implement ranking_section renderer to format Top20 scores and diagnostics in Markdown
- Develop AutoFinQuantStep for deterministic ETF ranking using TuShare data, Polars,
and a custom extremely randomized tree ensemble
- Integrate quantitative rankings into backtest and portfolio analysis steps and reports
- Extend auto_fin pipeline with new quant_enabled and quant_required config options
- Enforce ranking constraints like unique codes, contiguous ranks, and normalized fusion weights
- Update analysis YAMLs with rules limiting data freshness, universe, and ranking usage
- Incorporate ranking outputs into all major markdown report bodies in Auto Fin pipeline
- Add concurrency-limited asynchronous TuShare client to fetch required market data
- Introduce cross-sectional rank correlation and NDCG metrics for ranking quality evaluation
* feat(auto_fin): implement stage-wise notification and reporting for analysis pipeline
- Refactor notification config in daily_cookbook.yaml to support dispatch steps
- Update AutoFinNotificationStep to deduplicate notifications per run stage
- Add _notify_stage method in pipeline to send notifications for each analysis stage
- Implement persistence and notification for event, backtest, US correlation, and portfolio stages
- Modify pipeline flow to persist reports and notify after each stage completion
- Adjust metadata to track notifications and errors per stage
- Update tests to verify stage-wise notification sending and deduplication
- Remove older combined report persistence in favor of modular stage handling
* feat(auto_fin): add outbound proxy support for Tushare API usage
- Introduce BaseOutboundProxy reference in AutoFinPipelineStep and AutoFinQuantStep
- Update TushareResearchClient and trade calendar fetch to accept and use proxy URL
- Create _ProxiedTushareApi adapter to route Tushare requests via explicit HTTP proxy
- Modify create_tushare_api utility to optionally return proxied API client
- Add unit tests covering proxy forwarding and client behavior with managed proxies
- Ensure proxy usage respects explicit proxy URL over environment fallback
- Integrate outbound proxy into data fetching and quantitative research steps
* feat(auto_fin): enforce checkpoint time validation and add state models
- Introduce AnalysisState base class and specific states for event, backtest, and US correlation analyses
- Replace analysis output types with corresponding state classes in run schemas
- Add require_checkpoint_reached method to validate decision_at/data_cutoff against current time
- Enforce checkpoint time checks before analysis steps in event, backtest, portfolio, and quant analyses
- Refactor quant data loading to include adjustment factors and apply price adjustments without fallback
- Update analysis YAML docs to require real-time checkpoint validation and forbid using future data
- Improve portfolio run serialization by excluding redundant legacy fields and nested proposed actions
- Add helper to extract readable sections from persisted checkpoint documents
- Fix event analysis output validation to reject events and sources with future timestamps
* feat(auto_fin): auto-select latest reached checkpoint if none specified
- Extend checkpoint config to accept empty string for auto selection
- Add static method to compute latest checkpoint reached by current time
- Modify pipeline step to auto-select checkpoint based on trade calendar and time
- Adjust force flag default depending on whether checkpoint is explicit or auto
- Log details when checkpoint is auto-selected to improve observability
- Add comprehensive tests for auto checkpoint selection logic and edge cases
- Remove deprecated default and required constraints from force parameter in config
* refactor(auto_fin): unify datetime comparison with compare_datetimes utility
- Replace direct datetime comparisons with compare_datetimes function calls
- Use cmp_to_key with compare_datetimes for sorting datetime tuples and lists
- Update validation logic in backtest, event, analysis, and ledger modules for consistent datetime handling
- Add unit tests to verify handling of naive and aware datetime comparisons in event and backtest validations
- Ensure marked_at and interval_end timestamps are set and compared consistently using compare_datetimes
- Improve correctness of ordering and conditional checks related to timestamps throughout auto_fin steps and ledger code
* feat(auto_fin): add datetime comparison helper for mixed timezone data
- Implement compare_datetimes function to handle naive and aware datetimes
- Ensure naive datetime is interpreted in the known timezone of the counterpart
- Facilitate comparisons between legacy and timezone-aware Auto Fin data
- Add module docstring explaining purpose of the helpers
* docs(auto_fin): enforce unique ETF representative per sub-theme in analysis rules
- Update backtest.yaml to recommend or highlight only one ETF per sub-theme for ETF analyses
- Modify event.yaml to map only one representative ETF per sub-theme, avoiding duplicate recommendations
- Revise portfolio.yaml to restrict holdings/buys to a single ETF per sub-theme, preventing repeated buys of highly overlapping ETFs
- Adjust us_correlation.yaml to retain only one representative A-share ETF per sub-theme for mapping or recommendation
- Add test to verify presence of new sub-theme uniqueness guidance in step prompts
* feat(auto_fin): separate draft model and include deterministic fusion ranking
- Introduce _PortfolioProposalDraft pydantic model for agent-authored fields before ranking
- Discard any "fusion_ranking" data from draft to prevent conflicts with canonical ranking
- Modify AutoFinPortfolioStep to receive draft, enrich with fusion_ranking, and produce final output
- Update tests to use _PortfolioProposalDraft and validate deterministic fusion ranking propagation
- Add async test verifying fusion ranking is correctly set in portfolio output with no errors
* refactor(auto_fin): rewrite and simplify Auto Fin schema and steps
- Remove legacy Auto Fin analysis step modules and helpers
- Replace complex ranking and portfolio models with simplified current-news models
- Update schema to focus on news-case workflow with new domain models
- Remove A-share decision checkpoints and backtest details from schema
- Simplify recommendation and decision output structures
- Clean up deprecated state and utility functions
- Update Auto Fin steps initialization to new pipeline steps only
- Improve uniqueness validation for themes and ETFs in research plan
* feat(auto_fin): implement full local cache and analysis workflow for Auto Fin
- Add AutoFinDataStep to prepare and cache daily TuShare data with lookback
- Add AutoFinAnalysisStep to analyze cached data and generate Markdown report
- Implement detailed time window, ETF filtering, and historical case validation
- Introduce YAML prompts for planning and decision-making steps
- Update .gitignore to include reme_workspace/
- Clean up config and import structure for auto_fin steps
- Remove old pipeline.py and consolidate functionality into new modules
- Use polars for efficient CSV reading and data processing
- Ensure atomic writes and strict JSON serialization for cache files
- Enforce rules on news timing, ETF universe, and historical case usage
* fix(auto_fin): restrict news data source to '财联社' in analysis and cache
- Update analysis templates to specify current news as from '财联社' only
- Modify news fetching functions to filter by source '财联社'
- Add validation method to check cached news source correctness
- Update news caching logic to exclude non-'财联社' news
- Enhance unit tests with multiple sources to ensure filtering works
- Confirm news API calls include source filter parameter as '财联社'
* refactor(auto_fin): convert I/O methods to asynchronous implementations
- Change _news, _dataset, and _theme_data methods to async for improved concurrency
- Move JSONL and CSV reading operations to asynchronous wrappers using asyncio.to_thread
- Remove synchronous _read_jsonl and _read_csv functions, integrate them as static async class methods
- Update cache validation methods to async, awaiting I/O operations accordingly
- Adjust usage of dataset and news retrieval in analysis step to await asynchronous methods
- Add async unit test to validate JSONL reading with unicode line separators
- Preserve existing functionality while enabling non-blocking file and data access
* fix(nx_file_graph): defer networkx import and improve dependency handling
- Move networkx import inside NxFileGraph constructor for lazy loading
- Raise ImportError with original exception context if networkx is missing
- Remove module-level fallback assignment of nx to None
- Expand test to block loading of multiple optional core dependencies eagerly
- Change exception type in test from ModuleNotFoundError to AssertionError
- Update test comments to reflect broader optional dependency checks
* feat(embedding_store): add quota retry delay mechanism for embedding requests
- Introduce quota_retry_delay parameter to configure wait time before retry on quota exhaustion
- Implement detection of insufficient quota errors in LocalEmbeddingStore without external SDK
- Add retry logic with custom delay when quota is insufficient during embedding requests
- Update configuration to set max_retries and quota_retry_delay defaults for embedding store
- Add unit tests covering quota exhaustion retry behavior with delay and opt-in control
- Ensure existing retry behavior remains unchanged if quota_retry_delay is not set
* feat(auto_fin): add detailed logging to analysis and data fetching steps
- Add _preview static method for bounded diagnostic output in analysis.py
- Log prompt start, completion, errors, and validation details in _reply method
- Add info logs for major processing steps in execute method of analysis.py
- Add debug and info logs for cache validation, data fetching, and pagination in data.py
- Log conditions for skipping reports and cache plans in data.py execute method
- Log download summaries and cache writes for news and ETF data
- Improve error logging with exception details in cache validation functions
- Ensure all logs include context such as record counts, paths, and parameters
* refactor(auto_fin): overhaul Auto Fin workflow and schema contracts
- Replace old Auto Fin schema models with comprehensive new data classes
- Remove legacy Auto Fin analysis step in favor of modular agent-based steps
- Introduce AutoFinAgentStep for validating structured agent replies
- Simplify data cleaning and JSONL writing utilities for news cache
- Remove synchronous and asynchronous dataset methods from analysis step
- Redefine Auto Fin analysis configuration for 360-day news retention and multi-step pipeline
- Remove embedded analysis prompt templates and replace with agent-driven logic
- Update __init__.py exports to match new step implementations and remove deprecated classes
- Improve error handling and validation in agent step reply processing
- Clean up redundant imports and unused code in analysis and data preparation modules
* feat(auto_fin): add detailed logging for analysis and data processing steps
- Add timing logs to measure agent prompt processing duration in analysis.py
- Log news cache hits and news write paths with record counts in data.py
- Include detailed info logs for news download start and completion in data.py
- Add start, progress, and completion logs with topic and event counts in history.py
- Log start and completion of merge step including path and ETF count in merge.py
- Add start and done logs with window and news counts in topic.py
* feat(auto_fin): enhance schema and steps with detailed ETF and event modeling
- Replace and add multiple AutoFin schema classes to support detailed ETF selection,
historical research, market analysis, forecast models, and report output with validation
- Implement Shanghai timezone normalization and strict validation in schema models
- Remove deprecated AutoFin analysis agent step and consolidate reply handling in base step
- Introduce AutoFinStep base class with shared helpers for prompt handling, data fetching,
logging, and JSONL file operations
- Add AutoFinDataStep to manage daily news data complete with schedule validation, caching,
and source validation logic
- Update cookbook configuration to customize auto_fin step parameters and simplify
outbound proxy settings
- Refactor imports and clean unused code for better maintainability
* feat(auto_fin): introduce detailed historical event resolution and market similarity analysis
- Add AutoFinHistoricalEventReference and AutoFinHistoricalSimilarity models for refined event referencing and similarity judgment
- Implement validation to ensure non-empty critical fields and uniqueness of historical news IDs
- Develop method to resolve Agent-selected historical event references from workspace files with strict path and existence checks
- Enrich historical events with market entry and future returns data after resolution
- Redesign market step to calculate similarity-weighted ETF forecasts based on matched historical event similarities
- Enforce validation on matched historical events for uniqueness and proper weight summation
- Simplify merge step output to final Markdown report without YAML frontmatter and redundant fields
- Update user instructions for history search, market, and merge steps to reflect new data structures and responsibilities
- Adjust test suite to cover new schema and step behavior changes, including enhanced validation and JSON output formats
* feat(auto_fin): add new cron jobs and output analysis jsonl
- Add new cron jobs auto_fin_1145_cron and auto_fin_1800_cron with auto_fin_steps
- Change auto_fin_0930_cron schedule to run Monday to Sunday
- Extend merge step to write analysis data to auto_fin_analysis.jsonl
- Update unit tests to verify new cron jobs and their steps configuration
* fix(auto_fin): improve atomic file write and refresh daily index
- Change temporary file naming to include UUID for uniqueness and hidden prefix
- Replace atomic write method from using Path.replace to os.replace with safe unlink
- Add import and use os.replace for safer file replace operation
- Refresh daily index after writing auto finance markdown and JSONL files
- Import and call refresh_day_index in merge step to update file index asynchronously
* docs(cookbook): add optional SSH proxy configuration in README files
- Introduce optional SSH proxy setup in auto-fin and daily_paper cookbooks
- Provide instructions to enable outbound proxy via `daily_cookbook.yaml` and environment variables
- Add `REME_PROXY_IP` and `REME_PROXY_ACCOUNT` environment variables descriptions in multiple README files
- Update English and Chinese README and README_ZH documents with proxy details
- Maintain consistent formatting of environment variable tables across documents
* fix(file_io): include schema_version in hidden metadata keys
- Added "schema_version" to _INDEX_HIDDEN_METADATA_KEYS in _daily_index.py
- Updated _render_notes_block to always include additional keys regardless of schema_version
fix(deps): move pproxy dependency to later in pyproject.toml
- Removed pproxy from early dependencies list
- Added pproxy back near the end of dependency list for better ordering
fix(outbound_proxy): require pproxy package for ssh_http proxy
- Added importlib.util check for pproxy package presence
- Raise RuntimeError if pproxy is not installed when using SSH HTTP outbound proxy
- Improved error message suggests installing reme-ai with 'core' extra
* docs(readme): update News section with new Cookbook workflows
- Clarify introduction of optional Cookbooks with Daily Paper and Auto Fin workflows
- Update English README to reflect both paper discovery and file-native ETF event research
- Revise Chinese README to include financial news and historical market data research capability
- Maintain announcement of paper acceptance at Findings of ACL 2026
* feat(auto_fin): add calculation results to final Markdown output
- Implement _calculation_results to summarize forecast for each ETF analyzed
- Include program-calculated results in the JSON input for the Markdown report
- Update YAML template to incorporate calculation results and adjust recommendation rules
- Refine recommendation logic to rely on event impact judgments combined with calculation outputs
- Modify tests to verify presence of calculation results and updated report content and format
* up prompt
* fix(keyword_index): ignore non-indexable chunks during keyword sync
- Add is_indexable method to base and BM25 keyword index classes to check text tokenizability
- Update local file store to exclude non-indexable chunks from expected document IDs to prevent rebuild
- Fix JSONL chunker to correctly handle Unicode line separator U+2028 inside JSON strings without splitting
- Add test to ensure non-empty but non-indexable chunk does not trigger keyword index rebuild
- Add test to verify U+2028 character does not cause incorrect JSONL record splitting
* feat(daily-paper): add daily paper cookbook workflow with schema and tests
- Introduce daily paper schema types (DailyBriefOutput, PaperInfo, PaperNoteOutput, etc.)
- Create daily paper cookbook module with analyze, collect, digest, rank, and select steps
- Add cookbook entry point and integrate into main steps module
- Replace job config export with daily brief output in schema exports
- Add comprehensive unit tests covering pipeline, filtering, and output generation
- Update dependencies including openai-codex and pypdf packages
- Configure standalone daily paper cron job with proper scheduling and routing
* test(daily_paper): update tests to use Claude Code wrapper exclusively
- Add test to verify web search is disallowed by default in Claude Code
- Update imports to include DailyBriefOutput, PaperNoteOutput, and PaperSelection schemas
- Change test name from standalone_config_has_backend_split to reflect Claude Code only usage
- Remove default agent wrapper and configure all steps to use Claude Code wrapper
- Rename select_wrapper to cc_wrapper for clarity and consistency
- Remove duplicate Claude Code wrapper initialization
- Update test assertions to verify output schema usage matches expected sequence
- Remove unused as_llm component from standalone configuration test
* refactor(agent-wrapper): simplify skill resolution logic across all wrappers
- Replace duplicate skill resolution code with centralized _resolve_project_skills method
- Add project_path property with configurable relative path resolution
- Introduce proper validation for skill names and directory existence
- Change Codex wrapper to use project_path instead of workspace_path for skills
- Add SKILL.md requirement validation for project skills
- Remove redundant skill processing logic from individual wrappers
* feat(daily_paper): add daily paper workflow with PDF analysis and brief generation
- Implement shared state management and file helpers for daily-paper steps
- Add PDF download and text extraction capabilities with arXiv integration
- Create paper collection step with Hugging Face weekly/monthly rankings
- Build ranking system using reciprocal-rank fusion with memory keyword scoring
- Add Claude Code integration for paper analysis and detailed note generation
- Implement digest step to create final five-minute brief from detailed notes
- Add configuration for standalone daily cookbook application with cron scheduling
- Create typed schema for paper information, selection, and output formats
- Add atomic file writing with temporary file safety mechanisms
- Implement exclusion logic for previously recommended papers and daily filters
* feat(daily_paper): add DingTalk notification integration and enhance logging
- Integrate DingTalk markdown send step to notify groups about daily paper briefs
- Add comprehensive logging throughout daily paper workflow including start/finish events
- Update daily paper analysis prompt to include code repository context requirement
- Configure DingTalk notification in daily_cookbook.yaml with app credentials
- Add dingtalk-stream dependency for proactive message API integration
- Enhance daily paper README with DingTalk notification section and updated flow chart
- Implement detailed logging for each step including paper processing and agent calls
- Add test coverage for DingTalk markdown sending functionality and configuration
- Update pre-commit config to exclude skills directory from checks
- Add .claude/skills to gitignore for local development environment
* refactor(dingtalk): move dingtalk_stream import to local scope and improve code safety
- Moved global dingtalk_stream import to local scope in send.py to avoid eager loading
- Added dynamic import with error handling for optional dependency cases
- Updated test suite to verify lazy loading behavior works correctly
- Fixed markdown title generation by using safe variable naming in wait.py
- Enhanced test coverage for arxiv PDF download caching functionality
- Updated application context initialization with proper resource directory configuration
- Modified paper metadata to include source PDF path reference in output files
* refactor(daily_paper): remove manifest system and store selection metadata in digest files
- Remove JSON manifest creation and storage functionality
- Store selection data directly in digest file frontmatter instead of separate manifest files
- Add load_saved_selection method to rebuild selection from digest and paper-note metadata
- Update README documentation to reflect new cookbook workflow architecture
- Modify test cases to verify selection metadata in digest files instead of manifest JSON
- Remove unused json import from multiple daily paper modules
- Integrate PaperSelection schema for proper data validation in stored metadata
* docs(daily_paper): add bilingual cookbook guides
* feat(config): add environment variable configuration for agent subprocesses
- Add environment field to ApplicationConfig to store variables for agent subprocesses
- Remove dynamic loading of .env files in agent wrappers
- Introduce subprocess_environment property in base agent wrapper
- Pass application-level environment variables to Claude Code and Codex agents
- Load environment variables once at startup and pass to ReMe application
- Remove dependency on load_env utility in agent wrapper implementations
- Update tests to use configured environment instead of dynamic loading
- Remove unused environment loading utilities and related test cases
* refactor(mcp): remove channel notification system and related components
- Removed channel notification step implementation
- Removed claim channel step implementation
- Removed ChannelSink class from MCP service
- Removed channel-related documentation from AGENTS.md
- Removed channel instruction text from MCP service
- Removed all channel-related tests
- Updated application context metadata comment to remove channel sink reference
- Removed channel module initialization and imports
* feat(service): add job whitelisting capability to BaseService
- Add optional jobs parameter to BaseService.__init__ to configure job whitelist
- Store jobs as set in self.jobs attribute for efficient lookup operations
- Modify add_jobs method to filter jobs based on whitelist configuration
- Update documentation in both English and Chinese to describe new feature
- Add comprehensive unit tests for job whitelisting behavior
- Implement flowchart update showing new filtering logic
- Preserve existing enable_serve flag behavior alongside new whitelisting
* refactor(service): enhance service job validation and MCP tool injection
- Add strict validation for service jobs whitelist with detailed error messages
- Implement injected job arguments support for MCP services with conflict detection
- Add tool error handling for unsuccessful responses in MCP services
- Remove duplicate job names in Codex agent wrapper using dict.fromkeys
- Update MCP server argument format from single JSON array to repeated --job flags
- Add comprehensive test coverage for job injection and error handling scenarios
- Update documentation to reflect service job validation and MCP features
- Ensure application cleanup occurs even when service lifespan encounters errors
* feat(agent): update skill handling to preserve existing Claude skills
- Change skills parameter processing to use 'all' instead of filtered list
- Add logic to select project skills without restricting Claude's existing skills
- Update variable naming from 'skills' to 'selected_skills' for clarity
- Modify application context metadata documentation to clarify in-memory state usage
- Add test case to verify configured skills are added without filtering existing skills
- Update internal skill directory handling to use renamed variable consistently
* refactor(agent): restructure agent wrapper components and session storage
- Move CcFileSessionStore to separate module for better organization
- Add SDK package version logging in base agent wrapper
- Update Claude Code agent to use new session store structure with project keys
- Refactor Claude Code agent wrapper to use proper type hints and SDK integration
- Add support for server tool use events in Claude Code message processing
- Improve error handling and resource cleanup in streaming operations
- Update Codex agent wrapper with proper type annotations and configuration
- Remove deprecated system prompt mode handling from Claude Code wrapper
- Fix session path construction for Claude Code transcript storage
- Update dependency injection and configuration handling patterns
* fix(cc_agent_wrapper): resolve Claude Code SDK integration issues
- Added dataclass import and created _BlockState for content block metadata tracking
- Implemented proper MCP server name constant and tool context ID validation
- Fixed tool_context_id injection to prevent duplicate assignment errors
- Resolved skills parameter handling in build_options method
- Enhanced job tools integration with MCP servers mapping validation
- Replaced deprecated block_ids/block_types/tool_call_names with block_states dict
- Updated message_delta to emit USAGE chunks instead of REPLY_END
- Fixed stream result handling to ensure proper REPLY_END emission
- Improved error handling for session mirror failures and rate limits
- Added proper cleanup for expected trailing errors in streams
- Refactored Codex agent wrapper initialization and configuration management
- Removed obsolete system_prompt_mode from default config
- Enhanced test coverage for new block state and error handling features
- Fixed async generator handling with aclosing context manager
- Improved chunk type mapping for Claude Code SDK events
* refactor(tests): remove demo config tests from config parser test suite
- Removed test_demo_config_registers_llm_jobs function and its assertions
- Eliminated verification of LLM demo job configurations
- Removed checks for agent wrapper component settings
- Deleted assertions for model configurations and parameters
- Cleaned up deprecated test cases related to demo config parsing
* refactor(evolve): simplify Claude Code session store path structure
- Removed redundant project key subdirectory from session link generation
- Updated CcFileSessionStore initialization to use direct session directory path
- Maintained existing session layout compatibility for backward compatibility
- Added unit tests to verify session persistence behavior with existing transcripts
- Ensured UUID-based session files remain accessible at expected locations
- Preserved existing session directory structure without additional nesting
* refactor(agent): defer optional Codex SDK imports until first use
- Moved openai-codex imports inside functions to avoid mandatory dependencies
- Added TYPE_CHECKING guard for development time type checking only
- Implemented lazy loading mechanism with _get_async_codex_class function
- Updated AsyncCodex initialization to occur on demand rather than at module level
- Maintained backward compatibility while improving import performance
- Added test case to verify package import works without optional Codex SDK
- Updated agentscope dependency to version 2.0.4.post1 in pyproject.toml
* test(embedded): add compatibility tests for in-process ReMe embedding
- Add test suite for QwenPaw-style embedded configurations
- Verify optional defaults remain preserved in embedded configs
- Ensure in-process application API stays compatible
- Test model injection and lifecycle management compatibility
- Remove obsolete hermes agent plugin tests
- Update CLI import test to cover multiple optional SDKs
- Block claude_agent_sdk and openai_codex during import testing
* feat(benchmark): add golden answer validation and session review for LongMemEval
- Introduce GoldenCheckStep to validate LongMemEval golden answers using structured verdicts
- Add SessionReviewStep to extract query/answer-relevant evidence from all sessions
- Implement concurrent session processing with configurable concurrency limits
- Create check_golden job configuration with lme_review and lme_judge agent wrappers
- Add Qwen3.7-plus model configuration for enhanced processing capabilities
- Include python_execute tool integration for agent-based reasoning and date validation
- Generate comprehensive JSON output with session summaries and validation verdicts
- Add run_check_golden.py script for batch processing across all LongMemEval samples
- Configure proper logging initialization with console and file output options
- Update component registry and file I/O modules to support new benchmark features
* feat(scripts): add script to summarize LongMemEval check_golden verdicts
- Parse check_golden.json files across all LongMemEval samples
- Calculate accuracy metrics for golden answers and session IDs
- Provide breakdown by question type with percentage calculations
- Add command line options for listing bad samples and JSON output
- Include progress tracking showing completed vs pending samples
- Display confidence scores and date sanity checks statistics
* refactor(benchmark): move golden check scripts to longmemeval directory
- Moved run_check_golden.py from scripts/ to benchmark/longmemeval/
- Moved stats_check_golden.py from scripts/ to benchmark/longmemeval/
- Updated path resolution to use parents[2] instead of parent.parent
- Added new --list-run-failed option to stats script
- Added logging directory constant and functions for tracking launched samples
- Enhanced stats output with launched count and run failure information
- Improved error reporting with run failure details and log file paths
* feat(benchmark): add LongMemEval agentic answer workflow with session extraction
- Add LmeAgenticAnswerStep, LmeAutoMemoryStep, and LmeExtractSessionStep to __init__.py
- Create shared helper render_with_source for displaying search results with session_id
- Implement agentic_answer step with vector_search, bm25_search, and extract_session_by_id tools
- Add auto_memory step to convert each session into search-friendly daily notes
- Create extract_session step to retrieve and analyze raw session content by session_id
- Update jinli_lme.yaml with auto_memory, vector_search, bm25_search, and agentic_answer jobs
- Configure lme_memory, lme_extract, and lme_agentic_answer agent wrappers
- Enhance search steps with include_source option to show session_id metadata
- Add proper session_id tracking and collision handling in daily note generation
* feat(benchmark): add LongMemEval agentic answer evaluation pipeline
- Add session_id tracking to agentic_answer.py result metadata
- Introduce run_agentic_answer.py driver for complete pipeline execution
- Implement auto_memory, update_index, and agentic_answer job orchestration
- Add concurrent execution with configurable limits and staggering
- Create aggregation script for collecting tool-call trails and results
- Add stats_agentic_answer.py for comprehensive result analysis
- Implement resume capability with existing output detection
- Generate aggregate.json with per-sample breakdown and tool call summaries
* feat(steps): add ClearPathsStep for cleaning workspace outputs before rebuild
- Introduce ClearPathsStep to remove stale workspace files/directories
- Add support for specifying paths and config_keys as targets to clear
- Implement safety checks to prevent deletion of files outside workspace
- Add logging for cleared paths and warnings for invalid paths
- Configure clear_paths_step in jinli_lme.yaml to clean daily_dir
- Add clear_paths_step to clean mem_answer.json before rebuilds
* feat(benchmark): add resume functionality to agentic answer runner
- Replace --force flag with --resume flag for controlling job execution
- By default every job reruns with clean rebuild behavior using config clear steps
- Add --resume option to skip samples whose output already exists and continue interrupted batches
- Update documentation to reflect new default clean rebuild behavior
- Modify job skipping logic to honor resume flag instead of force flag
- Update dry-run output to show correct todo jobs based on resume status
- Change default example command to use --resume for continuing interrupted runs
* feat(benchmark): generate JSONL output for check golden records
- Add write_check_golden_list function to create JSONL file
- Write all readable check_golden records as JSONL format
- Include check_golden_list path in stats output
- Display generated JSONL file path in summary report
- Maintain UTF-8 encoding with non-ASCII character support
* refactor(benchmark): rename answer judge step and integrate LME LLM judge
- Rename AnswerJudgeStep to LmeLlmJudgeStep and update imports
- Add new llm_judge configuration in jinli_lme.yaml
- Update run_agentic_answer.py to include llm_judge in pipeline
- Modify LmeLlmJudgeStep to read from query.json and answer.json
- Write LLM judgement results back to mem_answer.json
- Add command line options for start/end sample range selection
- Update aggregate.json generation to include LLM judgement data
- Add resume capability for llm_judge job based on judgement presence
* refactor(benchmark): rename answer judge step and integrate LME LLM judge
- Rename AnswerJudgeStep to LmeLlmJudgeStep and update imports
- Add new llm_judge configuration in jinli_lme.yaml
- Update run_agentic_answer.py to include llm_judge in pipeline
- Modify LmeLlmJudgeStep to read from query.json and answer.json
- Write LLM judgement results back to mem_answer.json
- Add command line options for start/end sample range selection
- Update aggregate.json generation to include LLM judgement data
- Add resume capability for llm_judge job based on judgement presence
* feat(steps): add wait_for_paths_step to block until workspace files exist
- Introduce WaitForPathsStep class that polls for required workspace-relative paths
- Add step registration with 'wait_for_paths_step' backend identifier
- Implement path validation to ensure targets are within workspace boundaries
- Add polling mechanism with configurable intervals via poll_seconds parameter
- Include logging functionality with log_every_seconds parameter for status updates
- Add metadata tracking of waited paths and duration in response object
- Register step in index module and expose in public API
- Configure step in jinli_lme.yaml to wait for session_review.json before golden check
- Add script rename from run_check_golden.py to run_golden_check.py with enhanced options
* feat(benchmark): enhance longmemeval benchmarking with concurrency and progress tracking
- Add benchmark extra dependency group with portalocker requirement
- Introduce concurrent execution support for golden_check and session_review workflows
- Add progress reporting interval option with real-time status updates
- Implement global throttling mechanism for session review requests using file locks
- Enhance golden check validation with current schema verification
- Add active task tracking and graceful shutdown handling
- Rename check_golden scripts to golden_check for consistency
- Update statistics reporting with correct/incorrect terminology instead of reasonable
- Add stale format detection and compatibility handling for verdict fields
- Include both_correct rate calculation in accuracy metrics
- Add concurrency and staggering options for better resource management
* ci(workflow): add Windows smoke test workflow
- Create new workflow file .github/workflows/windows-smoke.yml
- Configure workflow to trigger on push and pull request events
- Set up Python environment with version 3.11
- Install package dependencies using pip
- Run version job as smoke test for CLI functionality
- Enable concurrency control to prevent duplicate runs
- Use matrix strategy for Python version testing
* feat(benchmark): add retry mechanism and health check for session review
- Added retry configuration options (retry_initial_seconds, retry_max_seconds, retry_max_attempts) to jinli_lme.yaml
- Implemented exponential backoff retry logic with configurable parameters in session_review step
- Added output_is_healthy function to verify session_review.json integrity and absence of failed reviews
- Updated resume functionality to skip only healthy outputs instead of all existing files
- Integrated JSON parsing and validation to check for failed reviews in output files
- Enhanced error handling and logging for retry attempts and recovery scenarios
* feat(benchmark): add LongMemEval session review statistics script
- Create stats_session_review.py to summarize session_review.json artifacts
- Add command line options for listing failed, missing, and run failed samples
- Implement JSON output mode for programmatic consumption
- Calculate and display health statistics including total samples, healthy outputs, failed sessions
- Provide detailed failure information with session IDs and error messages
- Generate re-run commands for samples with failed reviews
- Add percentage calculations for better statistical overview
- Include support for multiple output formats and detailed logging
* feat(benchmark): add LongMemEval output cleanup script and enhance golden check retry logic
- Added clean_sample_outputs.py script to remove generated LongMemEval files while preserving source inputs
- Implemented configurable retry mechanism in golden_check.py with exponential backoff strategy
- Added retry parameters (initial/max seconds and max attempts) to control failure recovery behavior
- Integrated asyncio support for asynchronous sleep during retry intervals
- Configured default retry settings in jinli_lme.yaml with 5s initial and 300s maximum intervals
- Preserved core files (query.json, answer.json, session/) while cleaning generated artifacts
* feat(benchmark): add AppleDouble file cleanup to sample output cleaner
- Remove AppleDouble files starting with '._' recursively including under session/
- Add is_under helper function to check if path is inside parent directory
- Track targets in set to avoid duplicate processing
- Include AppleDouble files in cleanup targets when not already covered by existing targets
- Maintain dry-run mode as default behavior with --apply flag for actual deletion
* refactor(benchmark): update LongMemEval sample output cleaning script
- Add time and Iterator imports for enhanced functionality
- Add --progress-every argument to control progress reporting frequency
- Replace is_under function with iter_sample_targets generator
- Implement detailed progress tracking with timing measurements
- Add sample-by-sample processing with elapsed time reporting
- Include AppleDouble file detection within session directory
- Update target counting and deletion statistics display
- Add conditional progress updates based on progress-every setting
- Improve dry-run mode with would-delete indication
* chore(benchmark): increase initial interval for session review step
- Changed START_INTERVAL_SECONDS from 1.0 to 3.0 seconds
- Adjusted timing parameters for better benchmark stability
* refactor(benchmark): implement coordinated retry mechanism for session reviews
- Add retry gate condition to coordinate concurrent review attempts
- Implement wait_for_healthy_start_slot to handle sequential retries
- Create mark_retrying and mark_recovered functions to track retry states
- Update reply_with_retry to accept index parameter for coordination
- Add has_prior_retry logic to prevent race conditions during recovery
- Ensure proper cleanup of retry state on success or failure
- Maintain backward compatibility while adding coordination features
* chore(benchmark): adjust session review start interval timeout
- Changed START_INTERVAL_SECONDS from 3.0 to 5.0 seconds
- Increased initial delay for session review benchmark step
- Updated timeout configuration for improved stability
* refactor(benchmark): update session review concurrency and throttling mechanism
- Replace global throttle with per-process concurrency control
- Add concurrency parameter with default value of 30 in config
- Add start_interval_seconds parameter with default value of 2 seconds
- Change default concurrency from 3 to 1 in command line interface
- Update documentation to reflect new throttling behavior
- Implement semaphore-based concurrency limiting for review tasks
- Modify retry mechanism to use local locking instead of global files
- Remove portalocker dependency for cross-process throttling
* refactor(config): update session review configuration and concurrency settings
- Removed deprecated retry configuration parameters from jinli_lme.yaml
- Increased MAX_CONCURRENCY from 30 to 60 in session_review.py
- Reduced START_INTERVAL_SECONDS from 2.0 to 1.0 in session_review.py
- Cleaned up redundant backend specifications in configuration file
- Simplified agent wrapper configurations by removing obsolete retry settings
* feat(benchmark): enhance LME auto memory step with advanced scheduling and error handling
- Add datetime parsing functionality for LongMemEval timestamps with regex pattern
- Implement configurable concurrency limits with MAX_CONCURRENCY of 60
- Introduce retry mechanism with exponential backoff for agent interactions
- Add session filtering based on date comparison with question_date validation
- Create rate limiting with start interval control between requests
- Implement sophisticated retry coordination using asyncio conditions
- Add comprehensive error tracking for failed and filtered session extracts
- Remove deprecated concurrency parameter from jinli_lme.yaml configuration
- Add structured output validation in session review step
- Include detailed metadata reporting with session statistics and errors
* fix(benchmark): adjust default concurrency for auto_memory job
- Changed default concurrency from 3 to 1 for auto_memory job to prevent API overload
- Updated help text to reflect new default value of 1 for concurrency parameter
- Modified documentation to clarify concurrency behavior varies by job type
* refactor(search): replace hardcoded candidate multiplier with constant
- Introduced _CANDIDATE_MULTIPLIER constant set to 10
- Replaced hardcoded factor of 5 with _CANDIDATE_MULTIPLIER in BM25 search
- Replaced hardcoded factor of 5 with _CANDIDATE_MULTIPLIER in vector search
- Updated test to verify both search steps use ten times limit for candidates
- Imported VectorSearchStep and Bm25SearchStep in test module
- Added comprehensive test case for candidate count calculation logic
* feat(lme): add data inspection error handling with fallback mechanism
- Implemented non-retryable data inspection error markers detection
- Added _is_data_inspection_error method to identify inspection failures
- Created fallback handling for data inspection errors in auto memory extraction
- Added fallback handling for data inspection errors in session review
- Extended failed extracts tracking with non-retryable and fallback flags
- Separated fallback extracts from regular failed extracts in reporting
- Enhanced error logging with specific data inspection failure messages
- Updated metrics to track fallback extractions and reviews separately
- Maintained existing retry logic for other exception types
* feat(benchmark): enhance session review statistics with fallback tracking
- Add support for identifying and listing non-retryable fallback reviews
- Introduce --list-fallback argument to display fallback review details
- Separate retryable failures from non-retryable fallbacks in reporting
- Track fallback samples and sessions separately from failed ones
- Update console output to show both retryable and non-retryable categories
- Include fallback details in JSON output with reasons and session info
- Modify failure counting logic to distinguish between retryable and fallback reviews
* feat(benchmark): add question_id tracking and enhanced fallback reporting
- Add question_id function to extract query.question_id from data
- Initialize question_id_by_id dictionary to store question IDs by index
- Store question_id for each sample during data processing
- Enhance fallback output to include question IDs and session information
- Format sample labels with question IDs when available
- Display session IDs associated with each fallback case
* feat(benchmark): add question_id support and improve bad sample reporting
- Add question_id_for function to extract question_id from multiple sources
- Add sample_label function to format samples as idx(question_id) when available
- Store question_id in data dictionary during processing
- Change bad_golden and bad_sessions to store full records instead of just indices
- Update list_bad output to show formatted labels with question_id information
- Improve error reporting with more detailed sample identification
* feat(benchmark): enhance golden check stats with structured output
- Add related_session_ids function to extract session IDs from verdict records
- Create grouped_records function to group records by question type
- Replace flat list output with JSON-formatted grouped records in list_bad option
- Replace flat list output with JSON-formatted grouped records in list_bad_sessions option
- Maintain Chinese labels while adding structured data presentation
- Improve readability of bad verdict record display with hierarchical grouping
* feat(benchmark): update data structure for question indexing
- Replace sample_label with _idx field for index tracking
- Add question_id field to store _question_id values
- Maintain backward compatibility with empty string defaults
- Preserve existing session_id functionality
- Update data mapping to include new fields in grouped results
* refactor(benchmark): streamline golden answer verification process
- Replace relevance filtering with comprehensive information extraction
- Remove is_relevant field and simplify session summary structure
- Change relevant_info to extracted_info for clarity
- Update golden check logic to work with full extractions instead of filtered summaries
- Simplify prompt instructions to focus on complete information extraction
- Remove redundant schema validation and structured output requirements
- Adjust statistics calculation to match new extraction approach
- Update metadata field names to reflect extraction rather than relevance checking
* feat(benchmark): add selective file deletion option to clean_sample_outputs
- Add --filename argument to delete only specific root-level files
- Modify iter_sample_targets function to accept optional filenames filter
- Implement validation for root-level filename constraints
- Update function calls to pass filenames parameter
- Add example usage for selective file deletion in documentation
* feat(benchmark): add error count metrics to golden check statistics
- Added golden_bad, session_bad, and both_bad calculation fields
- Updated console output format to include error counts per question type
- Modified table display to show both accuracy rates and error numbers
- Enhanced statistical summary with additional error breakdown metrics
* test(search): update search step tests with include_source parameter
- Added include_source=False parameter to VectorSearchStep initialization
- Added include_source=False parameter to Bm25SearchStep initialization
- Maintained existing RuntimeContext parameters for both search steps
- Updated test calls to match new constructor signature with include_source option
* refactor(embedding): update embedding model initialization and session storage paths
- Remove unused inspect import from as_embedding module
- Pass dimensions directly to embedding model constructor instead of using parameters
- Update session state file paths to use mem_session directory instead of resource
- Add mem_session_dir configuration option to application config schema
- Update workspace directory creation to include new mem_session directory
- Change AgentScope and Claude Code session paths to use mem_session directory
- Move embedding dimensions from parameters to top-level configuration
- Update AgentScope dependency version from 2.0.3 to 2.0.4
- Update integration tests to reflect new session file location paths
* chore(version): bump version to 0.4.0.8
- Update __version__ from 0.4.0.7 to 0.4.0.8 in __init__.py
* refactor(transfer): drop orphaned ingest step, make service discovery cross-platform
- Remove ingest step: superseded by auto_resource (drop files under
resource/ → watcher interprets them); its meta.json/<date>.md outputs
had no consumers and tripped the auto_resource watcher.
- Replace lsof/pgrep shell-outs in service_utils with psutil (per-process
enumeration, no root needed on macOS) for Windows/macOS/Linux support.
- Add cross-platform test coverage for _pid_on_port / _scan_reme_procs.
- Deps: +psutil, -filelock (only used by the removed ingest lock).
* chore(release): bump version to 0.4.0.5
* fix(mcp): resolve circular import issues and update dependencies
- Moved fastmcp imports inside functions to prevent circular dependencies
- Replaced _TRANSPORT_MAP with _VALID_TRANSPORTS set for transport validation
- Updated version number from 0.4.0.3 to 0.4.0.4
- Added claude-agent-sdk dependency to core optional dependencies
- Used TYPE_CHECKING imports for FastMCP related types
- Restructured transport mapping logic within function scope
- Fixed string annotation for CallToolResult type hints
* refactor(tests): update date handling in daily steps tests
- Replace _date.today() with timezone-aware now function
- Use Asia/Shanghai timezone for date formatting
- Change return format to use strftime instead of isoformat
- Import now function from reme.steps.evolve module
* refactor(tests): clean up unused imports in daily steps test
- Removed unused date import from datetime module
- Removed redundant pathlib Path import that was already imported later
- Kept necessary imports for asyncio, os, tempfile, warnings, and frontmatter modules
* test(daily_steps): update test to include application context for daily list step
- Add ApplicationContext initialization with temporary workspace directory
- Register file store component in application context
- Pass application context to DailyListStep constructor
- Maintain existing test assertion behavior for date metadata verification
- Changed project name in pyproject.toml from 'reme' to 'reme-ai'
- Updated dependency references in full extras to use 'reme-ai[core]' and 'reme-ai[dev]'
* refactor(file_chunker): replace file parser with file chunker component
- Rename file_parser module to file_chunker across codebase
- Update BaseFileParser to BaseFileChunker with corresponding component type
- Rename LinkedFileParser to MarkdownFileChunker for markdown-specific chunking
- Rename ChunkedFileParser to DefaultFileChunker for default byte-based chunking
- Update documentation references from file_parser to file_chunker
- Modify dependency injection in BaseStep to use file_chunker instead of file_parser
- Update configuration and component registration to use new chunker naming
- Rename all related test files and update test assertions accordingly
- Add recursive option to scan_store_changes_step in default configuration
* feat(database): enhance Neo4j connection with environment variable support
- Add support for NEO4J_PASSWORD environment variable as fallback
- Make password parameter optional in constructor with validation
- Update chromadb dependency from 1.3.5 to 1.5.7
- Configure CORS credentials based on origin settings
- Import os module for environment variable access
* feat(config): add timezone support and remove unused dialog directory
- Added timezone field to application config with IANA timezone support
- Removed unused dialog_dir configuration and related directory creation
- Replaced date.today() with timezone-aware now() function across daily operations
- Created evolve module with timezone-aware datetime functionality
- Updated daily_create, daily_list, and daily_reindex steps to use timezone-aware dates
* refactor(steps): update file chunker implementation
- Replace ChunkedFileParser with DefaultFileChunker in background steps
- Add module docstring to evolve steps package
- Update return type annotation to reflect new chunker class usage
* refactor(components): rename embedding and llm components to as_embedding and as_llm
- Rename reme4/components/embedding to reme4/components/as_embedding
- Rename reme4/components/llm to reme4/components/as_llm
- Update all imports and references from embedding to as_embedding
- Update all imports and references from llm to as_llm
- Change BaseEmbedding to BaseAsEmbedding and update inheritance
- Change BaseLLM to BaseAsLLM and update inheritance
- Update component types from LLM/EMBEDDING to AS_LLM/AS_EMBEDDING
- Update configuration keys from embedding/llm to as_embedding/as_llm
- Update all property references from llm to as_llm in step classes
- Update test assertions to use new component enum values
* refactor(embedding_store): rename embedding parameter to as_embedding
- Updated configuration key from 'embedding' to 'as_embedding'
- Renamed class attribute from 'embedding' to 'as_embedding'
- Updated method calls to use 'as_embedding' instead of 'embedding'
- Changed parameter name in constructor from 'embedding' to 'as_embedding'
- Updated documentation to reflect new parameter name
- Modified health check to use 'as_embedding' property
* feat(agent_wrapper): add unified agent wrapper component with multiple backends
- Introduce BaseAgentWrapper abstract base class for agent implementations
- Add AsAgentWrapper implementation using AgentScope framework
- Add CcAgentWrapper implementation using Claude Code SDK
- Register agent_wrapper component type in ComponentEnum
- Configure default agent_wrapper settings in default.yaml
- Implement tool integration for both AgentScope and Claude Code backends
- Support fluent configuration via set_system_prompt() and add_tools() methods
* feat(agent-wrapper): add structured output support for agent wrappers
- Import SystemMsg in AsAgentWrapper for structured output handling
- Add output_schema parameter support in AsAgentWrapper with generate_structured_output
- Implement set_output_schema method in BaseAgentWrapper for chaining configuration
- Add output schema support in CcAgentWrapper with JSON schema format option
- Return structured output when available in CcAgentWrapper response
- Refactor kwargs handling to use default values consistently across wrapper classes
* refactor(vector_store): make obvec and zvec vector stores optional dependencies
- Removed direct imports of ObVecVectorStore and ZvecVectorStore from init file
- Added try-except blocks for conditional importing of optional vector stores
- Updated error handling to check for both pyobvector and sqlalchemy in ObVecVectorStore
- Renamed _OBVECTOR_IMPORT_ERROR to _OBVEC_IMPORT_ERROR for consistency
- Moved pyobvector and related dependencies to optional 'obvec' extra
- Added separate 'zvec' optional dependency group
- Updated package configuration to exclude reme4 module patterns
- Removed reme4 entry point from console scripts
- Bumped version from 0.3.1.9 to 0.3.1.10
* refactor(dependencies): reorganize project dependencies and add optional seekdb support
- Move sqlite-vec, prompt_toolkit, and rich to earlier in dependencies list
- Remove pyseekdb from main dependencies and create separate seekdb optional dependency group
- Reorder pyyaml to later in the dependencies list
- Maintain all existing dependency versions while improving organization
* chore(deps): remove faiss-cpu dependency from pyproject.toml
- Removed faiss-cpu>=1.7.4 from the faiss dependency group
- Cleaned up unused faiss dependency configuration
- Updated project dependencies to exclude faiss-cpu package
* refactor(evolve): consolidate auto memory planner and writer into single step
- Removed separate AutoMemoryPlannerStep and AutoMemoryWriterStep classes
- Combined functionality into new AutoMemoryStep class in auto_memory.py
- Migrated prompt templates from separate YAML files to unified auto_memory.yaml
- Updated module imports to reference new consolidated step
- Simplified memory recording process using single ReAct agent instead of two-stage planning/writing
- Maintained same input/output contract with messages, session_id, and memory_hint parameters
- Preserved all original functionality for creating/updating daily notes with conversation facts
* fix(daily): update empty session_id handling to create day-level file
- Changed test to verify empty session_id creates day-level file daily/<date>.md
- Updated assertion to check response success instead of rejection
- Modified metadata verification to include path, session_id and created status
- Added file existence check for the generated daily markdown file
- Updated test name and print statement to reflect new behavior
- Fixed test registration to use updated function name
* fix(bm25_index): 修正BM25索引计算中的文档长度归一化问题
修复了在计算BM25相似度时对文档长度进行不正确归一化的bug，确保所有查询都能得到准确的相关性评分。
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* up
* refactor(steps): Rename and adjust indexing step logic
- Rename `scan_changes.py` and `reindex.py` to `clear_and_scan.py`
- Update implementation details of `ScanChangesStep` and `ClearAndScanStep`
- Modify the scheduling mechanism in `WatchChangesStep`
- Adjust step registration and parameter configuration in config files
- Update related tests to align with the new interface changes
* up
* feat(daily): replace daily CRUD operations with slug provisioning approach
* refactor(tests): migrate CRUD step tests from HTTP server to direct LocalFileStore
* up
* up
* up
* up
---------
Co-authored-by: huangsen <huangsen.huang@alibaba-inc.com>
* feat(file_store): add FAISS-backed local file store implementation
- Introduce FaissLocalFileStore class with vector search capabilities using FAISS IndexFlatIP
- Implement FAISS index persistence with binary format and JSON id-map sidecar
- Add automatic index rebuilding when sidecar files are missing or corrupted
- Support tombstone mechanism for efficient deletion and compaction
- Register 'faiss' component type in the registry system
- Add faiss-cpu dependency requirement to pyproject.toml
- Update configuration schema to use simplified parameter structure
- Enhance search step to support parameter override from runtime context
- Add comprehensive unit tests for FAISS store functionality
- Implement fallback to parent methods for basic CRUD operations
* refactor(search): simplify parameter retrieval logic
- Removed _param method that checked context and kwargs
- Directly use self.kwargs.get for all parameter retrievals
- Maintained same default values for vector_weight, candidate_multiplier, expand_links, and max_links_per_direction
- Reduced code complexity by eliminating redundant context checking logic
* feat(seekdb): add Seekdb file and vector stores with pyseekdb>=1.2.0
* refactor(seekdb): add pyseekdb_conn and remote-only host/port config
* refactor(embedding): remove env fallbacks from BaseEmbeddingModel; pass credentials in tests
* refactor(seekdb): drop tenant from client kwargs; default database test and empty password
* fix(deps): gate pyseekdb to Python >=3.11 for CI 3.10 compatibility
* fix(seekdb): satisfy pre-commit pylint and formatting for seekdb stores
* refactor(steps): Add job management methods and support registering them as tools
Added methods to the `BaseStep` class for retrieving, running, and registering jobs as tools, enhancing the functionality of the step class.
* fix doc
* chore(pyproject.toml): Update dependency versions and adjust package configuration
Bump agentscope version to 1.0.19 and reorganize the core dependency configuration structure.
* feat: add Neo4j file graph support and markdown parser with wikilink extraction
- Add Neo4jFileGraph implementation for property-graph storage with
virtual/real node handling and link management
- Introduce LinkedFileParser for markdown files with frontmatter,
wikilink graph extraction, and full-skeleton chunking
- Update pyproject.toml to include pyyaml, mistletoe, and neo4j
dependencies
- Modify .gitignore to exclude /vault and structure.md
- Change reme CLI entry point from reme_ai.main to remecli.reme
- Register new neo4j and md components in respective registries
* refactor(file-graph): add chunk_ids support to Neo4jFileGraph
Add chunk_ids field to File node properties in Neo4jFileGraph to
enable better content chunk tracking and management.
BREAKING CHANGE: File node schema now includes chunk_ids property
which may affect existing integrations.
feat(parser): implement wikilink resolution logic
Move path resolution logic from utils/path_resolver to
linked_file_parser module and enhance wikilink resolution with
folder-note rule support and improved error handling.
fix(tests): update test assertions and variable names
Update test cases to reflect changes in data structures and
variable naming conventions across various components.
chore(config): update package entry point reference
Change reme CLI entry point from remecli.reme:main to
reme_ai.reme:main in pyproject.toml.
refactor(utils): remove deprecated path_resolver module
Remove the old path_resolver utility module as its functionality
has been moved to linked_file_parser.
docs(file-graph): update Neo4jFileGraph documentation
Update class docstrings and comments to reflect new chunk_ids
property and other structural changes.
style(formatting): adjust code formatting and line breaks
Minor formatting improvements including line length optimization
and consistent spacing adjustments throughout the codebase.
* fix(pyproject.toml): correct entry point for reme command
Change the entry point from "reme_ai.reme:main" to "reme_ai.main:main"
to fix the module reference for the reme command in project scripts.
* feat(vector_store): add OceanBase as a VectorStore
* refactor(obvec): make it cleaner
* docs: add obvec related info
* refactor: minor update
* refactor: clean code and pass lint
* docs: remove unrelated edit
* docs: minor update
* feat(compactor): add extra instruction support and improve error handling
- Add extra_instruction parameter to compactor for custom guidance during message compaction
- Implement try-catch blocks around AS LLM initialization with detailed error logging
- Add extra_instruction parameter to ReMe.compact method with comprehensive documentation
- Update agentscope dependency from 1.0.17 to 1.0.18 in light installation
- Bump version number from 0.3.1.6 to 0.3.1.7
- Pass extra_instruction parameter through compactor instantiation and execution flow
* fix(core): add error handling for AS LLM formatters and token counters initialization
- Wrapped AS LLM formatters initialization in try-except blocks
- Added specific error logging for failed AS LLM formatter initialization
- Wrapped AS token counters initialization in try-except blocks
- Added specific error logging for failed AS token counter initialization
- Applied same error handling pattern to both initial setup and restart operations
- Maintained existing warning logs for unsupported backends
* refactor(core): replace text truncation utilities with new marker system
- Remove old truncate_text_utils module and its exports
- Replace TRUNCATION_MARKER_START with _TRUNCATION_NOTICE_MARKER constant
- Update as_msg_stat.py to split content using new marker format
- Modify FileIO tool to use TRUNCATION_NOTICE_MARKER for continuation hints
- Change is_truncated function checks to use marker presence detection
- Move transformers dependency from main deps to light extra dependencies
- Update tool result compactor tests to verify marker instead of is_truncated calls
* feat(file_io): enhance file operations with path resolution and append functionality
- Add expanduser() to resolve file paths with ~ symbol
- Implement proper file existence and type validation in update_file
- Add new append_file method to append content to files
- Update truncation notice format for better readability
- Fix typo in error message from "provide" to "provided"
- Update transformers dependency in pyproject.toml
- Remove duplicate transformers dependency from light extras
* refactor(file_io): disable pylint too-many-return-statements warning
* perf(file_watcher): increase default polling delay and optimize watcher configuration
- Increased default poll_delay_ms from 1000ms to 2000ms to reduce CPU usage
- Removed force_polling parameter as it's no longer needed with updated polling strategy
- Simplified async watch configuration by removing conditional force_polling logic
- Reduced overall system resource consumption during file watching operations
* refactor(memory): update conversation log documentation in memory summary
- Changed "Raw conversation logs" to "Earlier conversation logs" for clarity
- Added warning note about potentially large dialog file sizes
- Improved formatting with additional line break for better readability
- Maintained existing compressed summary integration unchanged
* feat(memory): add long-term memory support to file-based memory system
- Initialize _long_term_memory attribute as empty string
- Add memories section to content when long-term memory exists
- Consolidate summary and memories into single user message
- Format memories with markdown header # Memories
- Maintain existing compressed summary functionality
- Join multiple content parts with double newlines
* chore(deps): update version and move litellm to dev dependencies
- Updated package version from 0.3.1.2 to 0.3.1.3
- Removed litellm from main dependencies in pyproject.toml
- Added litellm as fixed version dependency in dev group
- Maintained litellm requirement while reorganizing dependency structure
* chore(deps): move litellm dependency to full extras
- Moved litellm==1.80.0 from main dependencies to full extra
- Kept litellm as optional dependency for users needing full feature set
- Maintains backward compatibility for light installation option
* feat(pyproject): add litellm dependency to project configuration
- Added litellm==1.80.0 as optional dependency in pyproject.toml
- Created new litellm extra group for LiteLLM integration
- Updated full dependency group to include the new litellm option
* update
* refactor(memory): remove unnecessary type check and update error logging
* refactor(core): standardize logger import and update agentscope dependency
* fix(memory): disable console output and add logging for summarizer component
* feat(core): replace OpenAI token counter with custom ReMe token counter
- Replace OpenAITokenCounter with ReMeTokenCounter implementation
- Add support for HuggingFace mirror and configurable tokenizer
- Register ReMeTokenCounter as default token counter in registry
- Update config to use hf backend with Qwen2.5-7B-Instruct model
refactor(memory): convert token counting methods to async in message handlers
- Change count_str_token, stat_message, count_msgs_token to async methods
- Update format_msgs_to_str and context_check to use async token counting
- Modify _format_tool_result_output to support async token counting
- Adjust all dependent methods to await async token counting calls
feat(memory): add dialog persistence to in-memory storage
- Implement _append_messages_to_dialog for saving messages to JSONL files
- Add dialog_path parameter to ReMeInMemoryMemory constructor
- Persist messages to daily JSONL files based on timestamp grouping
- Update mark_messages_compressed to save and remove compressed messages
- Modify clear_content to persist all messages before clearing memory
refactor(ops): update token counter type hints and initialization
- Change BaseOp to use HuggingFaceTokenCounter instead of TokenCounterBase
- Update type annotations for as_token_counter property and parameters
- Remove direct token counter injection from Compactor and ContextChecker
- Pass as_token_counter parameter through service context mechanism
style(logging): improve error logging with exception details
- Replace logger.error with logger.exception in browser control tool
- Change logger.error to logger.exception in memory get tool error handling
- Add proper exception logging with stack trace information
chore(config): add token counter configuration to light YAML
- Add as_token_counters section with default hf backend configuration
- Configure Qwen/Qwen2.5-7B-Instruct model with mirror support enabled
- Set up pretrained_model_name_or_path and use_mirror parameters
test(context): update context check tests to async implementation
- Convert verify_context_check_invariants to async function
- Update context check test methods to use async calls
- Change stat_message calls to await async implementation
- Modify test_empty_messages and test_below_threshold_returns_all to async
* feat(core): implement context checking and memory management features
* refactor(core): replace direct loguru import with logger utility function
* refactor(reme): remove RuntimeContext dependency and simplify context checking
* feat(docs): add raw conversation persistence to ReMe framework
- Add Weikang Zhou as a contributor in pyproject.toml
- Update README.md with new author in software citation
- Add new paper reference for AgentscopeReMe framework
- Include arXiv link and publication details
- Add full author list for the research paper
- Update bibliography with proper formatting
