25 commits

Author	SHA1	Message	Date
xyf2020	f04eedb3ab	feat(Tag filter): Add an optional rebuildable tag index for file frontmatter (#517) ... * Add optional tag generation and normalization to auto memory * Add tag index components and clean up temporary JSONL files * Preserve tag index state when reconciliation fails * Refactor and streamline application implementation * Fix pylint C1803 warnings in tag normalization tests * Document optional tag index configuration * Make tag index failures non-blocking and disable auto-memory tags * fix(tag-index): fail closed and support reindexing * fix(tag-index): preserve complete query expressions --------- Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>	2026-09-07 15:52:17 +08:00
WQS	5c17874f73	feat(auto_resource): interpret image resources into daily notes (#500) ... * feat: add auto_image step for image resource caption notes * feat: wire image resources into the resource watch loop * refactor: split auto resource processors behind router * refactor: align resource processor module names * refactor: preserve auto resource compatibility * refactor: clarify auto resource routing structure * fix: address auto resource review concerns * test: scope auto resource fixtures * docs: align auto resource processor wording * test: cover image resize failures * fix: harden image resource lifecycle * fix: preserve resource image detail and linked daily ownership * style(file-graph): stabilize multiline docstring formatting	2026-09-07 12:00:38 +08:00
jinliyl	f5ec230fef	feat: add DSH memory integration and organize extensions (#461) ... * feat: add DSH memory integration and organize extensions * fix: support newer DSH release candidates * fix: address DSH integration review feedback * fix: handle DSH cross-day retry edge cases	2026-08-20 15:31:51 +08:00
jinliyl	d3aee1adf5	feat(evolve): report auto-dream file changes (#458) ... * feat(evolve): report auto-dream content changes * perf(evolve): use lightweight dream snapshots	2026-08-19 15:44:06 +08:00
jinliyl	215c1f72f2	feat: refine local-first research and memory workflows (#444) ... * feat: refine local-first research workflows * fix: delegate structured output tool choice * refactor(auto-fin): fetch and filter rolling CLS news * fix(auto-fin): keep imports portable across platforms * feat(auto-fin): expose CLS fetch controls * fix(auto-fin): propagate configurable news window * feat(auto_fin): normalize hybrid wikilinks in report body - Add _normalize_hybrid_wikilinks method to remove redundant Markdown destinations - Use regex to identify hybrid wikilinks with optional destinations - Replace redundant destinations with simpler wikilink format for clarity - Ensure normalization is failure-safe with exception handling and logging - Update report body normalization process to apply hybrid wikilink fix - Add unit tests to verify correct normalization and failure safety behavior * fix(dream): serialize integration with application-wide asyncio lock - Add application-wide asyncio.Lock to serialize digest writes during integration - Update _snapshot_digest to capture metadata per bucket - Validate bucket association when recovering from file changes - Add tests ensuring recovery only from the correct bucket - Add tests confirming integration lock is shared across application context - Enhance strict topic YAML loading validation in dream utils - Add tests for strict topic loading rejecting invalid or lossy fields * fix(cookbook): enable configurable job_tools for digest and merge steps - Update daily_cookbook.yaml to add job_tools: [memory_search, read] in digest steps - Modify DailyPaperDigestStep to read job_tools from kwargs instead of fixed list - Modify AutoFinMergeStep to similarly read job_tools from kwargs - Update tests to pass job_tools explicitly when invoking these steps - Remove hardcoded _TOOLS constants and replace with dynamic job_tools handling * fix: retry incomplete dream receipts * perf(pdf): increase max PDF pages limit from 20 to 35 - Updated configuration max_pdf_pages from 20 to 35 in daily_cookbook.yaml - Modified code to extract up to 35 pages instead of 20 in analyze.py - Updated README and README_ZH to document the increased max_pdf_pages - Adjusted unit test assertions to reflect new max_pdf_pages limit of 35 * fix memory integration and daily paper links * docs clarify cookbook tool usage	2026-08-11 23:32:34 +08:00
jinliyl	c5d92a24ab	feat: weave dream wikilinks into contextual prose (#428)	2026-08-06 17:40:01 +08:00
jinliyl	9218a2d0e3	refactor: derive dialog paths from session_dir (#421) ... * refactor: derive dialog paths from session directory * fix: normalize configured session paths * fix: align dialog watch paths with writers * fix: reject absolute session directories	2026-08-06 17:07:14 +08:00
xyf2020	5bc46c88b6	feat(benchmark): enhance session memory retrieval and isolate benchmark assets (#409) ... * chore(benchmark): isolate dataset/workspaces/results per benchmark - Move shared benchmark/{datasets,memory_workspaces,results} into per-benchmark subdirs benchmark/<name>/{dataset,workspaces,results} - Update beam/longmemeval config.yaml and run.py path defaults - Relocate longmemeval download.py to benchmark/longmemeval/ (downloads into dataset/ subdir); inline dataset download docs into README - Update .gitignore: benchmark/*/{dataset,workspaces,results}/ - Move result-{beam,longmemeval}.md to benchmark/results_md/ and drop result- prefix; update README links - Fix stale path refs in llm_judge.py and logs/demo_search_format.py * feat(benchmark): add read tool to agentic answer and update BEAM results - Add 'read' to job_tools in BaseAgenticAnswerStep for file reading capability - Document read tool usage in lme/agentic_answer.yaml system prompt - Update result-beam.md with latest evaluation scores (OVERALL: 0.623/0.580) * feat(auto_memory): add source line-number markers for note traceability - Add _format_history hook in AutoMemoryStep with line-number annotation - Override in BeamAutoMemoryStep to prefix each turn with [Ln] for citation - Add session_file variable to prompt templates for source marker paths - Simplify repeated extraction rules by referencing system prompt - Enhance agentic_answer search strategy (multi-search, read tool hint) - Add warning log on ReadStep failure * feat(beam): enhance auto_memory with source markers and pilot ingest tooling * refactor(beam): rename max_chunk_words to max_segment_words, drop one-off pilot scripts * feat: add CompressorStep and search_v2 dual-mode session compression - Add CompressorStep (reme/steps/evolve/compressor.py) for direct LLM text compression with optional query-guided relevance filtering - Extend search_v2_step to support query-aware and query-independent session transcript compression via _compress injected kwargs - Refactor _source_format.py: split into render_chunk_entries + join_chunk_entries; session chunks now render line-aligned with L<n>: prefixes for verbatim/compressed parity - Add JOB_TOOLS and INJECTED_JOB_KWARGS to BaseAgenticAnswerStep for per-subclass tool and parameter injection - LmeAgenticAnswerStep injects _search._compress payload to enable query-aware compression during benchmark evaluation - Record compression ablation results in result-longmemeval.md - Add unit tests for CompressorStep and search compression paths * refactor(compress): relax session compression to lenient format-preserving strategy and update LME results * refactor(benchmark): make session compression config-driven via compress_session flag Move session-transcript compression from LME hard-coded injection to a runtime context flag set by evaluation.compress_session in each benchmark config. Compression is off by default for both BEAM and LME, and BaseAgenticAnswerStep now conditionally injects the _search compress payload only when the flag is truthy. * feat(lme/auto_memory): add source attribution markers with line numbers Add _format_history to annotate each turn with [Ln] line numbers and expose {session_file} in prompts so the agent can emit bare wikilink-style source markers like [[session/dialog/s1.jsonl#L1-L2,L5-L6]] at the end of factual entries. Consolidate the per-prompt body/format rules into references to the system prompt to avoid drift, and add frontmatter- protection guidance for the edit tool. * feat: improve agentic answer prompt and update beam 100K results - Strengthen abstention rule: prohibit extrapolation from related but non-direct evidence - Add multi-angle search after preliminary answer to check for conflicting/supplementary/updated information - Add max-iteration fallback to 'Information not found' - Update beam.md with 100K results (agentscope 2.0.4.post1, from scratch) including per-type token consumption and memory construction stats - config.yaml: 100K dataset, 20 workers for BEAM evaluation - run.py: add memory construction token usage tracking (default agent) - Overall: 0.635 → 0.654 (+0.019), contradiction_resolution: 0.338 → 0.478 (+0.140), abstention: 0.500 → 0.525 (+0.025) * feat(read): add session-aware formatting for read tool and update BEAM eval - Add truncate_session_output in _file_io.py to render jsonl session lines as [speaker @ time] content before byte-budget truncation - Add read_step_format_session flag to ReadStep, honoring injected job kwargs (precedence) and YAML fallback - Inject read_step_format_session=True into BaseAgenticAnswerStep so agentic answer reads render session transcripts human-readably - Refine BEAM agentic_answer prompt: continue multi-angle search after preliminary answer, forbid fabrication/extrapolation - Update BEAM config to 1M variant and add sequential 100K-eval / 1M-build shell script - Refresh benchmark/results_md/beam.md with latest results * chore(config): disable expand_links in beam and lme search_v2 configs * refactor(beam): drop one-off sequential 100K-eval-then-1M-build script * fix(benchmark): add compressor job to beam config and fix BEAM clone instructions - Add compressor job and compressor as_llm component to reme/config/beam.yaml (aligned with lme.yaml) so that compress_session: true works for BEAM - Add graceful degradation guard in search_v2._compress_session_entries: when the compressor job is missing from the active config, log a warning and skip compression instead of raising 'Job compressor not found'. Skipped when there is no app_context so unit tests mocking run_job still drive compression behavior. - Fix BEAM download instructions in README.md/README_ZH.md: add mkdir -p before cd benchmark/beam/dataset (the directory is gitignored and absent in a fresh clone) * fix(steps): guard compressor exceptions and fix ReadStep boolean override 1. search_v2: catch per-entry exceptions from run_job('compressor') inside compress() so asyncio.gather never propagates a compressor failure (e.g. temporary LLM outage). The failing entry keeps its original body while remaining entries are still compressed, preserving already-retrieved search results. 2. read: replace 'context_value or yaml_value' with an existence check so that a runtime-injected False can explicitly disable a YAML-true read_step_format_session flag. Add focused unit tests for both paths. * fix(search_v2): use existence check for strict_date_filter boolean override Replace 'context_value or yaml_value' with an existence-based check so that a runtime-injected False can explicitly disable a YAML-true strict_date_filter flag, consistent with the read_step_format_session fix. * refactor(search): simplify strict_date_filter fallback to truthiness-or * style(test): rename unused param to satisfy pylint W0613 --------- Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>	2026-08-05 19:23:42 +08:00
jinliyl	a9ec334adc	feat: simplify wikilink semantics and support line anchors (#412) ... * feat: simplify local links and support line anchors * fix: align line anchor tests with CI lint * fix: preserve local links across file moves * fix: encode markdown paths when rewriting links * refactor(read): keep explicit line range parameters * fix: simplify legacy link predicate compatibility * docs: align local link behavior with implementation * fix: skip unsupported markdown destination escapes * fix: normalize workspace link paths across platforms * fix: bound markdown link scanning * fix: keep local link processing linear * docs: clarify permissive markdown link parsing * fix: handle local link processing failures * refactor: limit file links to wikilink syntax * docs: align wikilink contract with implementation * fix: normalize dream and neighbor paths on Windows * fix: resolve workspace path for neighbor expansion	2026-08-05 11:47:50 +08:00
Sen Huang	550317c3bf	Revert "feat(plugin): add ReMe integration for Codex (#372)" (#400) ... This reverts commit a367c2ce13.	2026-07-29 18:14:05 +08:00
DiegoCluv7	a367c2ce13	feat(plugin): add ReMe integration for Codex (#372) ... * feat(plugin): add ReMe integration for Codex * fix(plugin): fix Codex plugin port, transcript ingestion, and Windows support * fix(plugin): correct Codex transcript schema, path validation, and hook fixes * test(plugin): add MCP round-trip tests * fix(plugin): rewrite parser and tests. * fix(plugin): reserve id-less messages, cover marketplace manifest, error handling, path fixes, and main sync Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-07-29 18:11:29 +08:00
xyf2020	f34dcdb09b	feat(Step tools): add white/black path prefix permission filtering to read, edit, write (#391) ... * feat(read): add white/black path prefix permission filtering to ReadStep * feat: add PrefixCheck mixin for path-prefix permission in file I/O steps * feat: add injected_job_kwargs mechanism and refine path-prefix permission * refactor(file_io): consolidate prefix_check into _path module	2026-07-27 17:20:21 +08:00
jinliyl	1687179f84	feat: add Auto Fin cookbook and managed outbound proxy support (#392) ... * feat: add ssh proxy * feat: add ssh proxy * feat: add ssh proxy * feat: add ssh proxy * feat: add prompt * feat: add agent wrapper * feat: add agent wrapper * feat: add agent wrapper * feat: add tushare skill * feat: add tushare skill * feat: add tushare skill * feat: add none stream * chore(deps): update dependency versions in pyproject.toml - Bump claude-agent-sdk from 0.2.123 to 0.2.126 - Upgrade pre-commit to version 4.6.1 or higher - Upgrade pytest to version 9.1.1 or higher * feat(agent_wrapper): add session compaction support and unify session commands - Introduce compact_session method to BaseAgentWrapper and implement it in AsAgentWrapper, CcAgentWrapper, and CodexAgentWrapper - Add session_command module with SessionCommandResult dataclass and handle_session_command function for /clear and /compact commands - Update __init__.py exports to include session_command handlers - Modify DingTalkWaitStep to handle session commands via handle_session_command function - Remove streaming mode from DingTalkWaitStep and simplify reply handling to final Markdown replies only - Add unit tests for session compaction methods and session command handling across wrappers and DingTalk integration - Clean up and remove obsolete streaming and card rendering code from DingTalk wait step - Adjust daily_cookbook.yaml to remove stream and card_update_interval config entries for DingTalk wait step * feat(auto_fin): add Auto Fin simulated portfolio cookbook workflow - Add comprehensive Auto Fin schema exports for multiple models and enums - Implement base class and helpers for Auto Fin analysis steps - Create file, state, and formatting utilities for Auto Fin with atomic file writes and locking - Define Auto Fin pipeline with four analysis agents: backtest, event, portfolio, and US correlation - Register Auto Fin package in cookbook workflows and schema initialization - Add detailed documentation in markdown describing the system design, workflow, and data contracts * feat(outbound_proxy): add application-scoped outbound HTTP proxy components - Introduce BaseOutboundProxy and OutboundProxyEndpoint as core contracts - Implement FixedHttpOutboundProxy for external HTTP proxy integration - Add SshHttpOutboundProxy providing SSH-backed local HTTP proxy tunnels - Register outbound proxy components in component registry and enumeration - Update components package to include outbound_proxy module - Add dependency on pproxy for SSH HTTP proxy bridging - Include comprehensive unit tests covering proxy lifecycle, validation, environment merging, error handling, readiness, and monitoring mechanisms * refactor(network): replace SSH proxy with explicit HTTP outbound proxy - Remove SSH proxy helper implementation and references in codebase - Add support for explicit HTTP proxy URL in arXiv and HuggingFace clients - Modify clients to use async context manager for consistent resource handling - Update daily paper steps to forward outbound proxy configuration explicitly - Change tests to cover new proxy usage model and remove SSH proxy mocks - Add outbound proxy component configuration in daily_cookbook.yaml - Ensure proxy URL usage disables environment trust in HTTP clients - Fix app context component enum access to be defensive against missing keys * feat(agent_wrapper): add managed proxy support for command environments - Introduce BaseOutboundProxy binding in BaseAgentWrapper for outbound proxy management - Add bash_environment and command_proxy_environment properties to apply proxy settings - Update WorkspaceBackend instantiation in AsAgentWrapper to use bash_environment - Inject managed proxy export commands into Claude Code Bash commands via hooks - Enhance CodexAgentWrapper to include managed proxy in shell environment policy - Modify daily_cookbook.yaml steps to specify outbound_proxy as default where needed - Add comprehensive unit tests verifying managed proxy injection and environment isolation - Ensure subprocess_environment remains unchanged while proxy is applied selectively to commands * refactor(memory): replace search job_tools with memory in daily cookbook config - Change workspace_dir default from .reme to reme_workspace - Replace search job_tools with memory across multiple components and jobs - Update descriptions to reflect long-term memory retrieval instead of search - Modify system prompts to instruct using memory for retrieving notes - Adjust unit tests to verify memory job_tools and job presence instead of search - Ensure consistency in configuration and tests for memory backend usage * refactor(config): rename memory to memory_search in daily cookbook config - Change all occurrences of "memory" to "memory_search" in job_tools and job definitions - Update related system prompts to reflect the new memory_search terminology - Modify unit tests to assert the presence of memory_search instead of memory - Ensure consistency across skills, job tools, and backend configurations in multiple components * feat(auto_fin): add deterministic quantitative research and ranking fusion - Introduce new schema models: EtfScore, RankingMetrics, ExtremeAnalysis, DimensionRanking, and FusionRanking to represent deterministic research outputs - Add ranking data to event, backtest, us_correlation, and portfolio analysis outputs - Implement ranking_section renderer to format Top20 scores and diagnostics in Markdown - Develop AutoFinQuantStep for deterministic ETF ranking using TuShare data, Polars, and a custom extremely randomized tree ensemble - Integrate quantitative rankings into backtest and portfolio analysis steps and reports - Extend auto_fin pipeline with new quant_enabled and quant_required config options - Enforce ranking constraints like unique codes, contiguous ranks, and normalized fusion weights - Update analysis YAMLs with rules limiting data freshness, universe, and ranking usage - Incorporate ranking outputs into all major markdown report bodies in Auto Fin pipeline - Add concurrency-limited asynchronous TuShare client to fetch required market data - Introduce cross-sectional rank correlation and NDCG metrics for ranking quality evaluation * feat(auto_fin): implement stage-wise notification and reporting for analysis pipeline - Refactor notification config in daily_cookbook.yaml to support dispatch steps - Update AutoFinNotificationStep to deduplicate notifications per run stage - Add _notify_stage method in pipeline to send notifications for each analysis stage - Implement persistence and notification for event, backtest, US correlation, and portfolio stages - Modify pipeline flow to persist reports and notify after each stage completion - Adjust metadata to track notifications and errors per stage - Update tests to verify stage-wise notification sending and deduplication - Remove older combined report persistence in favor of modular stage handling * feat(auto_fin): add outbound proxy support for Tushare API usage - Introduce BaseOutboundProxy reference in AutoFinPipelineStep and AutoFinQuantStep - Update TushareResearchClient and trade calendar fetch to accept and use proxy URL - Create _ProxiedTushareApi adapter to route Tushare requests via explicit HTTP proxy - Modify create_tushare_api utility to optionally return proxied API client - Add unit tests covering proxy forwarding and client behavior with managed proxies - Ensure proxy usage respects explicit proxy URL over environment fallback - Integrate outbound proxy into data fetching and quantitative research steps * feat(auto_fin): enforce checkpoint time validation and add state models - Introduce AnalysisState base class and specific states for event, backtest, and US correlation analyses - Replace analysis output types with corresponding state classes in run schemas - Add require_checkpoint_reached method to validate decision_at/data_cutoff against current time - Enforce checkpoint time checks before analysis steps in event, backtest, portfolio, and quant analyses - Refactor quant data loading to include adjustment factors and apply price adjustments without fallback - Update analysis YAML docs to require real-time checkpoint validation and forbid using future data - Improve portfolio run serialization by excluding redundant legacy fields and nested proposed actions - Add helper to extract readable sections from persisted checkpoint documents - Fix event analysis output validation to reject events and sources with future timestamps * feat(auto_fin): auto-select latest reached checkpoint if none specified - Extend checkpoint config to accept empty string for auto selection - Add static method to compute latest checkpoint reached by current time - Modify pipeline step to auto-select checkpoint based on trade calendar and time - Adjust force flag default depending on whether checkpoint is explicit or auto - Log details when checkpoint is auto-selected to improve observability - Add comprehensive tests for auto checkpoint selection logic and edge cases - Remove deprecated default and required constraints from force parameter in config * refactor(auto_fin): unify datetime comparison with compare_datetimes utility - Replace direct datetime comparisons with compare_datetimes function calls - Use cmp_to_key with compare_datetimes for sorting datetime tuples and lists - Update validation logic in backtest, event, analysis, and ledger modules for consistent datetime handling - Add unit tests to verify handling of naive and aware datetime comparisons in event and backtest validations - Ensure marked_at and interval_end timestamps are set and compared consistently using compare_datetimes - Improve correctness of ordering and conditional checks related to timestamps throughout auto_fin steps and ledger code * feat(auto_fin): add datetime comparison helper for mixed timezone data - Implement compare_datetimes function to handle naive and aware datetimes - Ensure naive datetime is interpreted in the known timezone of the counterpart - Facilitate comparisons between legacy and timezone-aware Auto Fin data - Add module docstring explaining purpose of the helpers * docs(auto_fin): enforce unique ETF representative per sub-theme in analysis rules - Update backtest.yaml to recommend or highlight only one ETF per sub-theme for ETF analyses - Modify event.yaml to map only one representative ETF per sub-theme, avoiding duplicate recommendations - Revise portfolio.yaml to restrict holdings/buys to a single ETF per sub-theme, preventing repeated buys of highly overlapping ETFs - Adjust us_correlation.yaml to retain only one representative A-share ETF per sub-theme for mapping or recommendation - Add test to verify presence of new sub-theme uniqueness guidance in step prompts * feat(auto_fin): separate draft model and include deterministic fusion ranking - Introduce _PortfolioProposalDraft pydantic model for agent-authored fields before ranking - Discard any "fusion_ranking" data from draft to prevent conflicts with canonical ranking - Modify AutoFinPortfolioStep to receive draft, enrich with fusion_ranking, and produce final output - Update tests to use _PortfolioProposalDraft and validate deterministic fusion ranking propagation - Add async test verifying fusion ranking is correctly set in portfolio output with no errors * refactor(auto_fin): rewrite and simplify Auto Fin schema and steps - Remove legacy Auto Fin analysis step modules and helpers - Replace complex ranking and portfolio models with simplified current-news models - Update schema to focus on news-case workflow with new domain models - Remove A-share decision checkpoints and backtest details from schema - Simplify recommendation and decision output structures - Clean up deprecated state and utility functions - Update Auto Fin steps initialization to new pipeline steps only - Improve uniqueness validation for themes and ETFs in research plan * feat(auto_fin): implement full local cache and analysis workflow for Auto Fin - Add AutoFinDataStep to prepare and cache daily TuShare data with lookback - Add AutoFinAnalysisStep to analyze cached data and generate Markdown report - Implement detailed time window, ETF filtering, and historical case validation - Introduce YAML prompts for planning and decision-making steps - Update .gitignore to include reme_workspace/ - Clean up config and import structure for auto_fin steps - Remove old pipeline.py and consolidate functionality into new modules - Use polars for efficient CSV reading and data processing - Ensure atomic writes and strict JSON serialization for cache files - Enforce rules on news timing, ETF universe, and historical case usage * fix(auto_fin): restrict news data source to '财联社' in analysis and cache - Update analysis templates to specify current news as from '财联社' only - Modify news fetching functions to filter by source '财联社' - Add validation method to check cached news source correctness - Update news caching logic to exclude non-'财联社' news - Enhance unit tests with multiple sources to ensure filtering works - Confirm news API calls include source filter parameter as '财联社' * refactor(auto_fin): convert I/O methods to asynchronous implementations - Change _news, _dataset, and _theme_data methods to async for improved concurrency - Move JSONL and CSV reading operations to asynchronous wrappers using asyncio.to_thread - Remove synchronous _read_jsonl and _read_csv functions, integrate them as static async class methods - Update cache validation methods to async, awaiting I/O operations accordingly - Adjust usage of dataset and news retrieval in analysis step to await asynchronous methods - Add async unit test to validate JSONL reading with unicode line separators - Preserve existing functionality while enabling non-blocking file and data access * fix(nx_file_graph): defer networkx import and improve dependency handling - Move networkx import inside NxFileGraph constructor for lazy loading - Raise ImportError with original exception context if networkx is missing - Remove module-level fallback assignment of nx to None - Expand test to block loading of multiple optional core dependencies eagerly - Change exception type in test from ModuleNotFoundError to AssertionError - Update test comments to reflect broader optional dependency checks * feat(embedding_store): add quota retry delay mechanism for embedding requests - Introduce quota_retry_delay parameter to configure wait time before retry on quota exhaustion - Implement detection of insufficient quota errors in LocalEmbeddingStore without external SDK - Add retry logic with custom delay when quota is insufficient during embedding requests - Update configuration to set max_retries and quota_retry_delay defaults for embedding store - Add unit tests covering quota exhaustion retry behavior with delay and opt-in control - Ensure existing retry behavior remains unchanged if quota_retry_delay is not set * feat(auto_fin): add detailed logging to analysis and data fetching steps - Add _preview static method for bounded diagnostic output in analysis.py - Log prompt start, completion, errors, and validation details in _reply method - Add info logs for major processing steps in execute method of analysis.py - Add debug and info logs for cache validation, data fetching, and pagination in data.py - Log conditions for skipping reports and cache plans in data.py execute method - Log download summaries and cache writes for news and ETF data - Improve error logging with exception details in cache validation functions - Ensure all logs include context such as record counts, paths, and parameters * refactor(auto_fin): overhaul Auto Fin workflow and schema contracts - Replace old Auto Fin schema models with comprehensive new data classes - Remove legacy Auto Fin analysis step in favor of modular agent-based steps - Introduce AutoFinAgentStep for validating structured agent replies - Simplify data cleaning and JSONL writing utilities for news cache - Remove synchronous and asynchronous dataset methods from analysis step - Redefine Auto Fin analysis configuration for 360-day news retention and multi-step pipeline - Remove embedded analysis prompt templates and replace with agent-driven logic - Update __init__.py exports to match new step implementations and remove deprecated classes - Improve error handling and validation in agent step reply processing - Clean up redundant imports and unused code in analysis and data preparation modules * feat(auto_fin): add detailed logging for analysis and data processing steps - Add timing logs to measure agent prompt processing duration in analysis.py - Log news cache hits and news write paths with record counts in data.py - Include detailed info logs for news download start and completion in data.py - Add start, progress, and completion logs with topic and event counts in history.py - Log start and completion of merge step including path and ETF count in merge.py - Add start and done logs with window and news counts in topic.py * feat(auto_fin): enhance schema and steps with detailed ETF and event modeling - Replace and add multiple AutoFin schema classes to support detailed ETF selection, historical research, market analysis, forecast models, and report output with validation - Implement Shanghai timezone normalization and strict validation in schema models - Remove deprecated AutoFin analysis agent step and consolidate reply handling in base step - Introduce AutoFinStep base class with shared helpers for prompt handling, data fetching, logging, and JSONL file operations - Add AutoFinDataStep to manage daily news data complete with schedule validation, caching, and source validation logic - Update cookbook configuration to customize auto_fin step parameters and simplify outbound proxy settings - Refactor imports and clean unused code for better maintainability * feat(auto_fin): introduce detailed historical event resolution and market similarity analysis - Add AutoFinHistoricalEventReference and AutoFinHistoricalSimilarity models for refined event referencing and similarity judgment - Implement validation to ensure non-empty critical fields and uniqueness of historical news IDs - Develop method to resolve Agent-selected historical event references from workspace files with strict path and existence checks - Enrich historical events with market entry and future returns data after resolution - Redesign market step to calculate similarity-weighted ETF forecasts based on matched historical event similarities - Enforce validation on matched historical events for uniqueness and proper weight summation - Simplify merge step output to final Markdown report without YAML frontmatter and redundant fields - Update user instructions for history search, market, and merge steps to reflect new data structures and responsibilities - Adjust test suite to cover new schema and step behavior changes, including enhanced validation and JSON output formats * feat(auto_fin): add new cron jobs and output analysis jsonl - Add new cron jobs auto_fin_1145_cron and auto_fin_1800_cron with auto_fin_steps - Change auto_fin_0930_cron schedule to run Monday to Sunday - Extend merge step to write analysis data to auto_fin_analysis.jsonl - Update unit tests to verify new cron jobs and their steps configuration * fix(auto_fin): improve atomic file write and refresh daily index - Change temporary file naming to include UUID for uniqueness and hidden prefix - Replace atomic write method from using Path.replace to os.replace with safe unlink - Add import and use os.replace for safer file replace operation - Refresh daily index after writing auto finance markdown and JSONL files - Import and call refresh_day_index in merge step to update file index asynchronously * docs(cookbook): add optional SSH proxy configuration in README files - Introduce optional SSH proxy setup in auto-fin and daily_paper cookbooks - Provide instructions to enable outbound proxy via `daily_cookbook.yaml` and environment variables - Add `REME_PROXY_IP` and `REME_PROXY_ACCOUNT` environment variables descriptions in multiple README files - Update English and Chinese README and README_ZH documents with proxy details - Maintain consistent formatting of environment variable tables across documents * fix(file_io): include schema_version in hidden metadata keys - Added "schema_version" to _INDEX_HIDDEN_METADATA_KEYS in _daily_index.py - Updated _render_notes_block to always include additional keys regardless of schema_version fix(deps): move pproxy dependency to later in pyproject.toml - Removed pproxy from early dependencies list - Added pproxy back near the end of dependency list for better ordering fix(outbound_proxy): require pproxy package for ssh_http proxy - Added importlib.util check for pproxy package presence - Raise RuntimeError if pproxy is not installed when using SSH HTTP outbound proxy - Improved error message suggests installing reme-ai with 'core' extra * docs(readme): update News section with new Cookbook workflows - Clarify introduction of optional Cookbooks with Daily Paper and Auto Fin workflows - Update English README to reflect both paper discovery and file-native ETF event research - Revise Chinese README to include financial news and historical market data research capability - Maintain announcement of paper acceptance at Findings of ACL 2026 * feat(auto_fin): add calculation results to final Markdown output - Implement _calculation_results to summarize forecast for each ETF analyzed - Include program-calculated results in the JSON input for the Markdown report - Update YAML template to incorporate calculation results and adjust recommendation rules - Refine recommendation logic to rely on event impact judgments combined with calculation outputs - Modify tests to verify presence of calculation results and updated report content and format * up prompt * fix(keyword_index): ignore non-indexable chunks during keyword sync - Add is_indexable method to base and BM25 keyword index classes to check text tokenizability - Update local file store to exclude non-indexable chunks from expected document IDs to prevent rebuild - Fix JSONL chunker to correctly handle Unicode line separator U+2028 inside JSON strings without splitting - Add test to ensure non-empty but non-indexable chunk does not trigger keyword index rebuild - Add test to verify U+2028 character does not cause incorrect JSONL record splitting	2026-07-25 18:09:39 +08:00
xyf2020	7b1da5a9ee	feat(benchmark): add BEAM & restructure LongMemEval evaluation framework (#375) ... * feat(eval): add LongMemEval evaluation framework with tool_defaults date injection - Add evaluation/longmemeval/ with run.py, config.yaml, and test scripts - Add reme/config/longmemeval.yaml for evaluation-specific model config - Add tool_defaults mechanism to as_agent_wrapper for injecting default tool kwargs (uses setdefault so LLM-provided values take priority) - Pass tool_defaults={'daily_write': {'date': day}} in auto_memory to ensure notes always use the correct historical date - Add timestamp interpolation (_interpolate_timestamps) in auto_memory for filling missing created_at fields via linear interpolation - Evaluation pipeline: ingest sessions -> dream -> search -> answer -> judge - Uses qwen3.6-flash for memory, qwen3.7-max for answer/judge * chore: gitignore logs/results/demo.py, keep empty dirs * chore: update .gitignore * feat(eval): add multiprocessing and session time filtering to longmemeval runner - Replace async execution with synchronous + multiprocessing for parallel item evaluation - Add filter_future_sessions option to only ingest sessions <= question date - Add question_types filtering in config - Add result summary with binary accuracy and avg score - Update config defaults (oracle variant, 50 items, 32 workers) - Minor code style fixes in agent_wrapper and auto_memory * feat: add bench_query_step with ReAct agent for benchmark query phase - Add BenchQueryStep using agent_wrapper with search job tool - Replace manual search+LLM answer in run.py with bench_query_job - Remove unused answer LLM config from longmemeval.yaml - Register benchmark step module in steps/__init__.py * feat: add start_date/end_date time filter support for search job - Add _extract_date_from_path to extract validated YYYY-MM-DD from chunk paths - Add start_date/end_date filtering in _matches_search_filter - Implement progressive recall in FaissLocalFileStore.vector_search - Promote start_date/end_date from context to search_filter in SearchStep - Add start_date/end_date parameters to search job in default.yaml - Add unit tests for date filter functionality * fix: validate/normalize date filters and harden _extract_date_from_path Address three code-review comments on the time_filter search feature: 1. Validate/normalize start_date and end_date before string comparison. _matches_search_filter does lexicographic comparison against path_date (always canonical YYYY-MM-DD). Raw caller values like '2026-2-28' or 'abc' would produce silently wrong results. Now SearchStep normalizes valid dates via extract_daily_date (with strptime fallback for non-zero-padded input) and silently ignores invalid dates with a logger.warning, removing them from the filter. 2. Clarify behavior for paths without embedded dates. Added optional strict_date_filter parameter (default False). When True and at least one date bound is active, chunks whose path yields no date (e.g. digest/personal/topic.md) are excluded. When False (default), the existing behavior is preserved — dateless paths pass through. 3. Harden _extract_date_from_path against non-standard suffixes. Previously parts[1].split('.')[0] accepted '2026-05-18.anything' as a valid date. Now only exact 'YYYY-MM-DD' (dir) and 'YYYY-MM-DD.md' (day-index) forms are accepted. * feat(eval): LLM-as-Judge per-type prompt routing, binary-only, progress tracking - Remove 0-5 score metric, keep only binary (yes/no) classification - Load per-question-type judge prompts from llm-as-judge.json (temporal-reasoning, knowledge-update, single-session-preference, __default__) - Replace SCORE_JUDGE_PROMPT with type-specific BINARY_JUDGE_PROMPT template - judge_response(): parameter 'metric' -> 'question_type', returns single 'judgment' - Summary output: add per-type accuracy breakdown, remove score stats - Add progress tracking: background thread prints PROGRESS every 10min - Add FINAL progress line and total elapsed time on completion - Add --log-level, --reme-log-level, -q CLI arguments - Parallel mode: pool.map -> pool.imap_unordered for real-time progress - config.yaml: full oracle (10000 items), 32 workers, all question types - Add kill.sh (process cleanup) and run_async.sh (background eval launcher) * docs: add LongMemEval oracle evaluation results (61.6% accuracy) * feat(bench): add MAX_ITERATION limit to BenchQueryStep and add _auto_memory.yaml * feat: add golden session benchmark & eval_only mode with refined prompt - Add benchmark/longmemeval/run_golden_session.py for golden session evaluation - Refine PROMPTED_SYSTEM_PROMPT: concise answer rule, remove 'Information not found' fallback - Add eval_only mode to run.py (--eval_only flag) - Add multiple eval config variants (evalonly, full, test5) - Add analyze_results.py for result parsing - Update auto_memory.yaml, longmemeval.yaml, application_config - Update result-longmemeval.md with latest evaluation results - Add benchmark results to .gitignore * update: refine answer prompts and increase max iteration to 6 - Tighten prompted-answer system prompt for more concise output - Comment out 'Information not found' fallback rule - Increase MAX_ITERATION from 5 to 6 in bench_query - Add recall_eval.py - Update evaluation results * feat(chunker): add dedicated JSON and JSONL file chunkers (cherry-pick from upstream #325) - Add JsonFileChunker: structure-aware chunking preserving nested key paths, optional list-to-dict conversion, size measured by json.dumps() char count - Add JsonlFileChunker: line-aligned sliding-window chunking with configurable overlap, supports char/byte mode switching - Register both chunkers in default.yaml (json for .json, jsonl for .jsonl) - Add comprehensive unit tests (21 + 20 test cases) * feat(service): add CLI service for local job execution (from upstream #334) - Introduce CliService to execute single jobs locally without serving ports - Add prepare_start_config and should_precheck_start functions for CLI job setup - Update reme start command to use CLI service when job argument is provided - Add show_metadata to client kwargs for optional CLI metadata output - Add unit tests for CLI service functionality and configuration handling * feat(steps): add BM25/vector search steps, Python execute step, and draft steps (from upstream #334) - Add Bm25SearchStep for plain BM25 keyword search with tool_context deduplication - Add VectorSearchStep for plain vector search with tool_context deduplication - Add PythonExecuteStep to run Python code in subprocess with timeout handling - Add AddDraftStep/ReadAllDraftStep for draft accumulation scoped by tool context - Update SearchStep with tool_context dedup, dynamic default limit via REME_SEARCH_LIMIT env, and candidate_multiplier default changed from 3.0 to 5.0 - Add comprehensive unit tests for all new steps * feat(search): add tool context deduplication and improve search configuration (#321) * feat(search): add tool context deduplication and improve search configuration - Modify _make_tool methods to accept and inject tool_context_id parameter - Add tool_context_id handling in AS and CC agent wrappers - Increase search candidate multiplier from 3.0 to 5.0 in default config - Extend HTTP client timeout from 30s to 3600s - Add tool context deduplication logic to prevent duplicate search results - Implement TTL-based expiration for seen chunks in tool contexts - Add comprehensive unit tests for tool context deduplication behavior - Update .gitignore to exclude longmemeval directory - Add time import for timestamp functionality in search step * refactor(search): replace time module with datetime for timestamp generation - Removed unused time import - Added static method _now_ts using datetime.timestamp - Updated clock parameter to use _now_ts method instead of time.time - Maintained same timestamp precision and functionality * fix(file_io): fix risk of out-workspace paths (#322) * fix(file_io): fix risk of out-workspace paths * chore(file_io): remove unused unittest file * fix(as_embedding): support both agentscope 2.0.2 and 2.0.3 (#323) 2.0.3 promoted `dimensions` to a required first-class constructor argument while keeping a backfill from `parameters.dimensions`; 2.0.2 has no such argument and reads `dimensions` from `Parameters`. Keep `dimensions` in `Parameters` for both versions and, when the model constructor accepts `dimensions`, pass `dimensions=None` so 2.0.3's backfill promotes it out of `parameters`. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * Bump version to 0.4.0.7 * refactor: delegate LLM-as-Judge to answer_judge_step and update eval config/results - run.py: replace inline judge logic with judge_response_via_job using app.run_job('answer_judge') - longmemeval.yaml: expand benchmark configuration - bench_query.py: enhance benchmark query step - result-longmemeval.md: update evaluation results - judge_all_plus_results.json: add judge all-plus results * refactor: split longmemeval.yaml into lme.yaml/beam.yaml and unify job names - Split reme/config/longmemeval.yaml into lme.yaml (LongMemEval) and beam.yaml (BEAM) - Unify job names across both configs: agentic_answer, answer_judge, context_answer - Update evaluation/longmemeval/run.py and evaluation/beam/run_beam_eval.py to use unified job names - Update all evaluation config YAMLs to reference lme.yaml - Add BEAM benchmark step implementations (agentic_answer, context_answer, llm_judge) - Remove obsolete config_test5.yaml and test_5sessions.py * eval: BEAM 100K & LongMemEval cleaned-S 评测结果记录 - BEAM 100K eval-only (32并发, 20 case): Agentic 0.631, Prompted 0.468 - LongMemEval final GT (500题): Agentic 89.0%, Prompted 83.6% - 新增 benchmark/result-beam.md, benchmark/result-longmemeval.md - benchmark/beam/config.yaml: num_workers=32 * refactor: restructure benchmark directory and clean up gitignore rules - Consolidate benchmark outputs to benchmark/results/ with .gitkeep - Remove old benchmark scripts, configs and result files from benchmark/beam/ and benchmark/longmemeval/ - Add datasets/README.md and datasets/README_EN.md with download instructions - Add datasets/longmemeval/download.py and final_groundtruth_cleaned_s.json - Add memory_workspaces .gitkeep placeholders - Restructure .gitignore: fix duplicate entries, add BEAM dataset exclusion, refine logs/results ignore patterns - Remove stale result-beam.md and result-longmemeval.md from project root * chore: clean up longmemeval benchmark scripts and update dataset docs - Remove obsolete longmemeval benchmark runner/stats scripts - Update datasets/longmemeval README and add Chinese translation - Clean up final_groundtruth_cleaned_s.json * docs(benchmark): add reproduction guide for LongMemEval and BEAM - Add bilingual README for benchmark runners (EN/ZH) - Cover prerequisites, dataset download, run commands, configs, outputs, logs, and kill.sh * refactor: migrate auto_memory steps from evolve to benchmark-specific modules - Split auto_memory into beam and lme benchmark-specific implementations - Add auto_memory.py and auto_memory.yaml under steps/benchmark/beam and steps/benchmark/lme - Slim down evolve/auto_memory.py and auto_memory.yaml to shared base only - Remove obsolete evolve/_auto_memory.yaml - Update benchmark run.py, config YAMLs, and step __init__.py registrations - Update llm_judge and context_answer minor adjustments - Remove outdated test_lme_final_answer_review.py * revert(as_agent_wrapper): sync with upstream/main Remove local-only comment to keep file identical with upstream/main. * style: add trailing commas in benchmark __init__.py __all__ lists * chore: disable vector_weight range assertion in SearchStep * chore: add tests/integration/logs/ to .gitignore * refactor: replace scipy.stats.kendalltau with pure numpy implementation scipy is not listed in project dependencies. Implement Kendall's tau-b rank correlation using only numpy to remove the undeclared dependency. * feat(benchmark): add binary score metrics, update BEAM 1M results, and improve LLM retry/prompt config - benchmark/beam/run.py: add binary score calculation per rubric item and per-type/overall binary stats - benchmark/beam/config.yaml: switch to 1M dataset, reduce workers to 18 - benchmark/result-beam.md: add 1M evaluation results with binary scores - benchmark/result-longmemeval.md: minor formatting - reme/config/beam.yaml: increase max_retries to 5 and add retry_delay 5.0 for all LLM components - reme/config/lme.yaml: increase max_retries to 5 and add retry_delay for judge/prompted/bench components - reme/steps/benchmark/lme/agentic_answer.yaml: improve search strategy and answer rules prompts * fix(benchmark): fix line-too-long and add pylint disable for main() * refactor(longmemeval): use single cleaned-S dataset with embedded ground truth - Switch to agentscope-ai/ReMe_longmemeval_clean_s_v2 HuggingFace source - Remove separate final_groundtruth_cleaned_s.json (ground truth now in data file) - Simplify download.py to fetch only longmemeval_s_reme_cleaned.json - Remove dataset.variant and dataset.ground_truth_path config options - Update benchmark and datasets READMEs to reflect new workflow - Update .gitignore for new dataset filename * fix: rename loop variable to avoid pylint redefined-outer-name warning * refactor(benchmark): restructure datasets/memory_workspaces into benchmark and simplify auto_memory steps * refactor(benchmark): extract BaseAgenticAnswerStep into base module - Add reme/steps/benchmark/base/agentic_answer.py with shared agentic answer logic - Refactor beam/lme AgenticAnswerStep to inherit from BaseAgenticAnswerStep - Simplify lme/context_answer.py and update context_answer.yaml - Update result-longmemeval.md with latest evaluation results (agentic 91.0%) * refactor(benchmark): remove context_answer steps and unused configs - Remove beam/lme context_answer job definitions and step implementations - Remove prompted LLM component from beam.yaml and lme.yaml - Delete jinli_lme.yaml (no longer needed) - Simplify benchmark run.py scripts - Clean up .gitkeep files and update .gitignore - Remove unused import in search.py * chore: remove benchmark/results/.gitkeep --------- Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135> Co-authored-by: jinliyl <6469360+jinliyl@users.noreply.github.com> Co-authored-by: imrewce <wce@pku.edu.cn> Co-authored-by: Sen Huang <48879559+ployts@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-21 19:09:50 +08:00
jinliyl	e7d44f6f3b	refactor(agent): unify agent subprocess env, sessions, skills, and MCP/service jobs (#382) ... * feat(config): add environment variable configuration for agent subprocesses - Add environment field to ApplicationConfig to store variables for agent subprocesses - Remove dynamic loading of .env files in agent wrappers - Introduce subprocess_environment property in base agent wrapper - Pass application-level environment variables to Claude Code and Codex agents - Load environment variables once at startup and pass to ReMe application - Remove dependency on load_env utility in agent wrapper implementations - Update tests to use configured environment instead of dynamic loading - Remove unused environment loading utilities and related test cases * refactor(mcp): remove channel notification system and related components - Removed channel notification step implementation - Removed claim channel step implementation - Removed ChannelSink class from MCP service - Removed channel-related documentation from AGENTS.md - Removed channel instruction text from MCP service - Removed all channel-related tests - Updated application context metadata comment to remove channel sink reference - Removed channel module initialization and imports * feat(service): add job whitelisting capability to BaseService - Add optional jobs parameter to BaseService.__init__ to configure job whitelist - Store jobs as set in self.jobs attribute for efficient lookup operations - Modify add_jobs method to filter jobs based on whitelist configuration - Update documentation in both English and Chinese to describe new feature - Add comprehensive unit tests for job whitelisting behavior - Implement flowchart update showing new filtering logic - Preserve existing enable_serve flag behavior alongside new whitelisting * refactor(service): enhance service job validation and MCP tool injection - Add strict validation for service jobs whitelist with detailed error messages - Implement injected job arguments support for MCP services with conflict detection - Add tool error handling for unsuccessful responses in MCP services - Remove duplicate job names in Codex agent wrapper using dict.fromkeys - Update MCP server argument format from single JSON array to repeated --job flags - Add comprehensive test coverage for job injection and error handling scenarios - Update documentation to reflect service job validation and MCP features - Ensure application cleanup occurs even when service lifespan encounters errors * feat(agent): update skill handling to preserve existing Claude skills - Change skills parameter processing to use 'all' instead of filtered list - Add logic to select project skills without restricting Claude's existing skills - Update variable naming from 'skills' to 'selected_skills' for clarity - Modify application context metadata documentation to clarify in-memory state usage - Add test case to verify configured skills are added without filtering existing skills - Update internal skill directory handling to use renamed variable consistently * refactor(agent): restructure agent wrapper components and session storage - Move CcFileSessionStore to separate module for better organization - Add SDK package version logging in base agent wrapper - Update Claude Code agent to use new session store structure with project keys - Refactor Claude Code agent wrapper to use proper type hints and SDK integration - Add support for server tool use events in Claude Code message processing - Improve error handling and resource cleanup in streaming operations - Update Codex agent wrapper with proper type annotations and configuration - Remove deprecated system prompt mode handling from Claude Code wrapper - Fix session path construction for Claude Code transcript storage - Update dependency injection and configuration handling patterns * fix(cc_agent_wrapper): resolve Claude Code SDK integration issues - Added dataclass import and created _BlockState for content block metadata tracking - Implemented proper MCP server name constant and tool context ID validation - Fixed tool_context_id injection to prevent duplicate assignment errors - Resolved skills parameter handling in build_options method - Enhanced job tools integration with MCP servers mapping validation - Replaced deprecated block_ids/block_types/tool_call_names with block_states dict - Updated message_delta to emit USAGE chunks instead of REPLY_END - Fixed stream result handling to ensure proper REPLY_END emission - Improved error handling for session mirror failures and rate limits - Added proper cleanup for expected trailing errors in streams - Refactored Codex agent wrapper initialization and configuration management - Removed obsolete system_prompt_mode from default config - Enhanced test coverage for new block state and error handling features - Fixed async generator handling with aclosing context manager - Improved chunk type mapping for Claude Code SDK events * refactor(tests): remove demo config tests from config parser test suite - Removed test_demo_config_registers_llm_jobs function and its assertions - Eliminated verification of LLM demo job configurations - Removed checks for agent wrapper component settings - Deleted assertions for model configurations and parameters - Cleaned up deprecated test cases related to demo config parsing * refactor(evolve): simplify Claude Code session store path structure - Removed redundant project key subdirectory from session link generation - Updated CcFileSessionStore initialization to use direct session directory path - Maintained existing session layout compatibility for backward compatibility - Added unit tests to verify session persistence behavior with existing transcripts - Ensured UUID-based session files remain accessible at expected locations - Preserved existing session directory structure without additional nesting * refactor(agent): defer optional Codex SDK imports until first use - Moved openai-codex imports inside functions to avoid mandatory dependencies - Added TYPE_CHECKING guard for development time type checking only - Implemented lazy loading mechanism with _get_async_codex_class function - Updated AsyncCodex initialization to occur on demand rather than at module level - Maintained backward compatibility while improving import performance - Added test case to verify package import works without optional Codex SDK - Updated agentscope dependency to version 2.0.4.post1 in pyproject.toml * test(embedded): add compatibility tests for in-process ReMe embedding - Add test suite for QwenPaw-style embedded configurations - Verify optional defaults remain preserved in embedded configs - Ensure in-process application API stays compatible - Test model injection and lifecycle management compatibility - Remove obsolete hermes agent plugin tests - Update CLI import test to cover multiple optional SDKs - Block claude_agent_sdk and openai_codex during import testing	2026-07-20 23:52:14 +08:00
Sen Huang	55ef4bd6ad	fix(proactive): expose topics in primary answer (#380)	2026-07-20 16:05:47 +08:00
jinliyl	c3b1e93918	feat(index): add file size limits and oversized file handling (#362) ... * feat(index): add file size limits and oversized file handling - Implement max_file_bytes configuration option for content processing jobs - Add default 20MB file size limit for background processing in default config - Skip oversized files during auto_resource step with appropriate metadata - Clear stale index entries when oversized files are modified - Add size-based filtering logic to update_changes step with skip reporting - Include file size validation in UpdateIndexStep with proper response handling - Add comprehensive tests for oversized file scenarios in auto_resource and update_index - Document file size limits in constants with appropriate thresholds * chore(version): bump version to 0.4.1.1 - Update __version__ from 0.4.1.0 to 0.4.1.1 in __init__.py * fix(index): isolate batch metadata and handle file races	2026-07-15 21:01:18 +08:00
Ziyang Guo	c060933e4d	fix(auto-memory): preserve message timestamps (#310) ... * fix(auto-memory): preserve message timestamps * fix(auto-memory): infer daily date from messages * feat(file_io): add strict date parsing and improve daily date handling - Add new parse_daily_date function for strict YYYY-MM-DD validation - Replace extract_daily_date with parse_daily_date for explicit date validation - Change _messages_day to use max date instead of min for historical imports - Reorder imports to maintain consistent module ordering - Move session message saving after date validation in auto_memory - Add comprehensive tests for invalid date rejection before saving - Add tests for strict YYYY-MM-DD date format validation - Update test names to reflect latest date behavior --------- Co-authored-by: Ziyang Guo <121015044+RunMarshal@users.noreply.github.com> Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>	2026-07-01 17:15:07 +08:00
Sen Huang	3dee10d4f9	feat: add Claude Code plugin with auto-memory functionality (#297) ... * feat: add Claude Code plugin with auto-memory functionality * refactor(auto_memory): fix spacing in json parsing logic	2026-06-26 14:46:42 +08:00
jinliyl	ffb4d08c4f	feat(mem): Enhance daily note system with metadata handling and write functionality (#295) ... * feat(file_io): add daily_write step for creating daily notes with conversation metadata - Add DailyWriteStep class that delegates to write job for creating daily notes - Register daily_write job in default configuration with proper parameters - Include validation for name and session_id path components - Add test coverage for daily_write functionality including metadata handling - Preserve existing job execution method in application.py after repositioning - Update base_step.py to use positional-only parameter syntax for job methods - Import and expose DailyWriteStep in file_io module initialization - Override reserved metadata keys (name, description, session_id, source_conversation) with fixed values - Refresh daily index after successful write operation - Generate proper source conversation links in markdown format * feat(daily): refactor daily note system with enhanced metadata handling - Introduce validate_filename_component function and export it - Add _INDEX_HIDDEN_METADATA_KEYS to hide conversation metadata from index - Update scan_notes to exclude hidden metadata keys from index rendering - Modify auto_memory to use daily_write tool and manage session frontmatter - Implement session note lookup and renaming based on frontmatter name - Update daily_list to return flattened note metadata including session info - Change daily_write to dispatch write step instead of running job - Add test cases for updated daily note functionality and metadata handling - Update version from 0.4.0.2 to 0.4.0.3 * fix(evolve): correct metadata update in auto memory response - Fixed trailing comma issue in metadata dictionary update - Ensured proper formatting of response metadata structure - Maintained existing functionality while fixing syntax error * refactor(auto_resource): replace daily_create with dynamic note management - Remove DailyCreateStep and related exports from file_io module - Replace static daily note creation with dynamic resource-linked card system - Implement LLM-suggested naming with frontmatter-driven file management - Add source_resource linking for tracking original files - Introduce collision handling with hash-based suffixes - Update documentation to reflect new resource card workflow - Modify auto_resource prompts to use write/edit tools instead of daily_create - Adjust test fixture comments to match new agent behavior - Update framework diagrams and quick start examples accordingly * feat(app): add version info to app initialization and update auto-memory logic - Include version number in application startup logging - Remove tool result truncation logic from auto-memory step - Update auto-memory to exclude tool_result blocks from saved history - Add test case to verify tool results are filtered out from message saving - Update YAML prompts to clarify filename naming rules without dates - Modify configuration to support new dispatch steps format with persistence control * feat(auto_memory): add note modification tracking and optimize frontmatter updates - Add _note_bytes and _note_modified methods to track actual file changes - Optimize frontmatter updates by checking existing metadata before update - Add modified flag to response metadata indicating actual note changes - Update logging to include modified status in various operations - Add comprehensive tests for modified/unmodified detection scenarios - Enhance result hook logic to skip when no actual changes occur - Refactor metadata handling to properly track creation vs modification status	2026-06-25 21:54:56 +08:00
jinliyl	8b82ff88d0	feat(evolve): enhance agent reply processing and logging capabilities (#293) ... * feat(evolve): enhance agent reply processing and logging capabilities - Add agent_reply_result_text function to extract final user-visible text from agent replies - Implement comprehensive logging throughout auto_memory, auto_resource, and dream modules - Add max_units configuration option to limit extracted memory units - Improve error handling and validation in auto_resource step - Refactor dream extract step to respect max_units limit during processing - Enhance summary rendering in dream finish step with detailed breakdown - Add result hook functionality for embedding hosts integration - Implement loose resource filename handling for root-level resources - Update test cases to reflect new functionality and improved error messages * test(background-steps): update fake upsert function to include created parameter - Modified fake_upsert function to accept 'created' parameter instead of '_created' - Added 'created' field to captured dictionary in fake_upsert function - Included 'created': True in the expected response dictionary for test case - Updated test assertion to match new parameter structure	2026-06-24 22:08:47 +08:00
jinliyl	7d86658f33	Refactor logging levels and add dream schema definitions (#291) ... * chore(logging): change info logs to debug level for data loading operations - Changed stopwords loading log from info to debug level - Changed file catalog nodes loading log from info to debug level - Changed file graph nodes loading log from info to debug level * feat(dream): add dream schema definitions and enum for auto-dream functionality - Add DreamBucketEnum with procedure, personal, and wiki values - Create comprehensive dream-related Pydantic models including DreamUnit, DreamTopic, DreamExtractOutput, IntegrateOutcome, TopicSelectionOutput, ProactiveResult, and DreamState - Move schema definitions from local step module to shared schema package - Update dream extraction and integration steps to use new enum-based bucket validation - Initialize digest directories for each dream bucket type - Enhance embedding store health check with workspace directory logging * refactor(tests): update DreamState import path in test_auto_dream.py - Move DreamState import from reme.steps.evolve.dream.schema to reme.schema - Maintain same functionality with updated module reference - Align import with new schema location in project structure	2026-06-24 16:44:03 +08:00
jinliyl	a3bd81bde2	Update version to 0.4.0.2 and improve tokenizer index handling (#290) ... * fix(core): update version number to 0.4.0.1 - Incremented version from 0.4.0.0 to 0.4.0.1 in __init__.py * fix(index): remove stopwords path from tokenizer config and add keyword index repair - Remove stopwords_path from tokenizer config to prevent index forking by install path - Add _sync_keyword_index_from_chunks method to repair keyword index when persisted state mismatches - Implement test for keyword index repair from persisted chunks when missing - Add test to verify tokenizer fingerprint ignores stopwords absolute path - Update version from 0.4.0.1 to 0.4.0.2 * feat(dream): add scan_days parameter to dream extraction process - Add scan_days configuration option to default.yaml with default value of 2 - Implement recent_dates utility function to calculate date ranges for scanning - Modify DreamExtractStep to scan multiple days based on scan_days parameter - Update dream extraction to process files across multiple dates instead of single day - Extend DreamState schema to include dates and scan_days fields - Update DreamTopicsStep to handle multi-day topic processing - Modify finish step to checkpoint files from all scanned dates - Add comprehensive tests for multi-day scanning functionality - Update prompt templates to include scan dates information - Refactor topics writing logic to target specific date rather than current date	2026-06-24 15:01:07 +08:00
Sen Huang	e31db5fe19	docs: rename vault_dir to workspace_dir in documentation and examples (#286) ... * docs: rename vault_dir to workspace_dir in documentation and examples * refactor(extract): format long method call across multiple lines * refactor(extract): format system prompt parameters for better readability	2026-06-22 16:58:57 +08:00
jinliyl	206a53e5ed	init: reme version 0.4.0 (#284)	2026-06-22 15:41:19 +08:00