12 commits

Author	SHA1	Message	Date
imrewce	354837f9af	feat(proactive): separate proactive refresh from auto dream (#488) ... * refractor(proactive): upgrade proactive feature with disentangled job and steps * refactor(proactive): apply audit fixes - rename read-side job 'proactive' -> 'proactive_read' (less confusing vs the refresh pipeline) - drop dedicated agent_wrapper.proactive; extraction reuses the default wrapper - simplify schema: remove unused ProactiveExtractOutput/TopicUpdate, drop resource_paths - extract no longer scans resource/ directly (daily notes already carry resource content) - update tests and docs accordingly * feat(proactive): strict extract-output gate and prompt total budget - parse_extract_reply now requires a contract section (follow_ups/extends/updates as a list); non-empty replies with misspelled section names trigger the existing one-shot retry instead of silently checkpointing changed files - pack_paths gains max_total_chars; extract packs newest daily material first, keeps the first file on overflow, and records omitted files in a trailer (default budget 300000 chars, configurable via max_total_chars) - tests: schema gate unit, schema-error retry e2e, budget unit + e2e * feat(proactive): add scenario-card plan step and generative agenda step * feat(proactive): digest-personal profile personalization and leaner LLM contract - extract/plan/agenda now draw a user profile block from <digest_dir>/personal/*.md (frontmatter description + body excerpt, per-file budget, profile.md fallback) - all daily access honours the configured daily_dir (prompt paths parameterized, config-driven fallbacks) so workspaces using e.g. memory/ work unchanged - schema trim: drop dead fields errors/material_paths, carry_forward_all -> count - shrink LLM output contract: new topics emit title/reason/confidence/paths only; keywords removed end-to-end, evidence derived from paths[0] (updates keep it) * fix(proactive): skip checkpoint when extract reply stays unusable after retry Two consecutive unparseable replies now short-circuit the round without checkpointing, so the same material is retried next round instead of being silently consumed (closes the residual audit #1 gap: the structural gate detected schema-wrong output but a double failure still checkpointed). * fix(proactive): replace running bool with reference-counted job activity tracker for the idle gate * refactor(proactive): remove job activity tracking and idle gate, restore job tree to upstream * fix(proactive): address second audit round (readonly reader, mtime checkpoint, wider fallbacks, profile containment, horizon content, expiry boundary) * refactor(dream): strip interests.yaml ownership from dream, proactive is now the sole writer * refactor(dream): separate proactive topic generation * ci: update renamed auto dream smoke test * fix(proactive): complete refresh migration and docs --------- Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>	2026-09-07 17:23:37 +08:00
jinliyl	513fb5b7f4	feat: extract Daily Paper into an independently packaged plugin (#491) ... * feat: extract Daily Paper into a plugin * fix: satisfy clean-environment quality checks * fix: address daily paper review feedback	2026-08-26 17:32:46 +08:00
jinliyl	618e8cec66	feat: add entry-point plugin system and extract Auto Fin (#459) ... * feat: add entry-point plugin system * fix: harden plugin config and client loading * docs(workflow): add detailed manual for publishing reme-auto-fin to PyPI - Provide step-by-step instructions for updating project.version and merging branches - Explain dependency verification for reme-ai on PyPI during build - Specify requirements for GitHub Actions secret configuration and version uniqueness - Describe manual workflow triggering and input of version number - Recommend publishing order for related projects - Clarify that only manual dispatch triggers publishing, no automatic triggers on push or tag * feat: support plugin-defined component types * refactor: simplify plugin configuration * fix: isolate plugin loading and defer client fallback * refactor: freeze built-in component registry * fix: isolate config entry point loading * fix: complete auto-fin package metadata	2026-08-19 17:23:23 +08:00
jinliyl	215c1f72f2	feat: refine local-first research and memory workflows (#444) ... * feat: refine local-first research workflows * fix: delegate structured output tool choice * refactor(auto-fin): fetch and filter rolling CLS news * fix(auto-fin): keep imports portable across platforms * feat(auto-fin): expose CLS fetch controls * fix(auto-fin): propagate configurable news window * feat(auto_fin): normalize hybrid wikilinks in report body - Add _normalize_hybrid_wikilinks method to remove redundant Markdown destinations - Use regex to identify hybrid wikilinks with optional destinations - Replace redundant destinations with simpler wikilink format for clarity - Ensure normalization is failure-safe with exception handling and logging - Update report body normalization process to apply hybrid wikilink fix - Add unit tests to verify correct normalization and failure safety behavior * fix(dream): serialize integration with application-wide asyncio lock - Add application-wide asyncio.Lock to serialize digest writes during integration - Update _snapshot_digest to capture metadata per bucket - Validate bucket association when recovering from file changes - Add tests ensuring recovery only from the correct bucket - Add tests confirming integration lock is shared across application context - Enhance strict topic YAML loading validation in dream utils - Add tests for strict topic loading rejecting invalid or lossy fields * fix(cookbook): enable configurable job_tools for digest and merge steps - Update daily_cookbook.yaml to add job_tools: [memory_search, read] in digest steps - Modify DailyPaperDigestStep to read job_tools from kwargs instead of fixed list - Modify AutoFinMergeStep to similarly read job_tools from kwargs - Update tests to pass job_tools explicitly when invoking these steps - Remove hardcoded _TOOLS constants and replace with dynamic job_tools handling * fix: retry incomplete dream receipts * perf(pdf): increase max PDF pages limit from 20 to 35 - Updated configuration max_pdf_pages from 20 to 35 in daily_cookbook.yaml - Modified code to extract up to 35 pages instead of 20 in analyze.py - Updated README and README_ZH to document the increased max_pdf_pages - Adjusted unit test assertions to reflect new max_pdf_pages limit of 35 * fix memory integration and daily paper links * docs clarify cookbook tool usage	2026-08-11 23:32:34 +08:00
jinliyl	d5e0d2837b	refactor: rebuild auto-fin and daily-paper cookbooks on structured-output agents (#432) ... * refactor: rebuild auto-fin and daily-paper cookbooks on structured-output agents Rework the auto-fin and daily-paper cookbooks to run on structured-output LLM agents instead of Claude Code agent wrappers, replace the SSH proxy with data-source mirrors, and rewrite the affected unit tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(auto_fin): unify JSON output serialization and writing - Extracted _write_output static method to serialize and write Pydantic models as compact JSON - Replaced inline JSON dump and write calls with _write_output usage across auto_fin steps - Added _report_path and _current_report for managing intra-day reports in AutoFinMergeStep - Updated auto_fin merge step to write output via new _write_output method - Enhanced news reading with caching in AutoFinHistoryStep - Refined returns calculation to handle events before close on non-trading days correctly feat(daily_paper): improve note path resolution and metadata handling - Introduced iter_note_metadata generator for safe Markdown frontmatter iteration - Added resolve_unique_note_path to avoid note filename conflicts on disk and in used titles - Updated analyze, collect, digest, and select steps to use centralized constants and helpers - Used utc_now_iso for consistent timestamping in metadata - Replaced direct frontmatter loads with iter_note_metadata in collect and analyze steps - Replaced hardcoded paper selection count with PAPER_COUNT constant in all relevant places - Added _MAX_SELECT_ATTEMPTS constant in select step for attempt management - Improved error messages for filename validation in daily paper title normalization feat(auto_fin): add multi-run cron schedules for intraday refinement - Defined three auto_fin cron jobs at 09:30, 11:30, and 18:00 Shanghai time for gradual report updates - Each intraday run adds evidence cumulatively instead of replacing prior output wholly - Updated daily_cookbook.yaml to register new cron schedules and remove legacy 12:00 cron refactor(auto_fin_data): clean ETF code handling and page limits - Replaced hardcoded DEFAULT_ETF_CODES with required non-empty config value "etf_codes" - Added constants for major news and fund page limits to control pagination - Improved ETF name extraction logic to handle missing fields consistently fix(auto_fin_merge): fix report retrieval and merging logic - Added support for getting current intra-day report in addition to previous day's report - Modified merge template to include prior and current report sections for better context - Adjusted report path handling to consistently use Path objects test(auto_fin): add coverage for returns calculation and report retrieval - Added test for returns when event occurs before close on non-trading day, checking next session entry - Added test for previous and current report retrieval feeding merge context with disk files - Extended test asserts for auto_fin cron schedule changes in config style(daily_paper): reorder and cleanup imports - Reorganized imports in _common.py for clarity and added missing collections.abc.Iterator import - Cleaned up commented and unused imports across daily_paper steps * feat: add configurable upstream mirror proxy * style: format auto-fin data step * fix: align cookbook mirrors and contracts --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-08-07 23:53:14 +08:00
jinliyl	eac8223387	feat: add frontend-ready wikilink graph APIs (#414)	2026-08-05 16:45:50 +08:00
xyf2020	6b035c6553	feat(evaluation): track job calls and agent token usage in benchmarks (#406) ... * feat(counter): extend counter tree utils and record job call statistics - replace global_counter_next with fetch-and-add style global_counter_add/inc, plus read-only global_counter_get and global_counter_get_all - record per-job call counts in app_context.metadata via BaseJob._record_call, covering background/cron/stream jobs - update agentic_answer step and utils exports; add unit tests for job counting and counter utils * feat(evaluation): add check_job_count interface and report search calls in benchmarks - Extract _counter_key from BaseJob._record_call for reusable counter lookup - Add reme.utils.evaluation_interface.check_job_count read-only helper - Track and report average search calls per query in beam and longmemeval benchmarks * job counter * token消耗量统计 * benchmark输出完整token消耗统计 * benchmark统计输出改用标准差 - beam/longmemeval 的工具调用与 token 统计由方差改为标准差输出 - 修复 lint: 局部变量遮蔽 importlib.metadata、补充测试 docstring - black 格式化 * fix(evaluation): preserve complete token usage metrics * fix: exclude stream replies from token accounting * Revert "fix: exclude stream replies from token accounting" This reverts commit 85bf32064d. * Reapply "fix: exclude stream replies from token accounting" This reverts commit 6722c24dc5. * support agent scope 2.0.5 * feat: support injection_config to disable runtime state injection in benchmarks - Add InjectionConfig passthrough in AsAgentWrapper.reply() - Disable inject_runtime_state in BaseAgenticAnswerStep to avoid wall-clock time conflicting with benchmark query_time anchors - Disable inject_runtime_state in beam/lme llm_judge calls * feat: agentscope dual-version compat & benchmark improvements - Add version_tuple utility for semantic version comparison - AsAgentWrapper: version-aware InjectionConfig, max_iters doubling, and token usage collection (reply vs reply_stream) for AS>=2.0.5/<2.0.5 - Default inject_runtime_state=False in wrapper to avoid benchmark time-anchor conflicts; remove per-callsite injection_config overrides - longmemeval run.py: support question_ids filter in dataset config - Fix unused import in test_evaluation_interface; format fixes * chore: remove temporary flip-test benchmark config * revert: pin agentscope to 2.0.4.post1 and drop dual-version compat * fix(evaluation): clarify usage semantics and atomic counters --------- Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135> Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>	2026-08-04 11:42:18 +08:00
jinliyl	2f79977df0	refactor(auto_fin): replace similarity with direction classification for historical events (#395) ... - Add AutoFinHistoricalDirectionReference model to classify historical events by direction - Remove AutoFinHistoricalSimilarity and related similarity score usage - Update AutoFinMarketSelection to handle same and opposite direction event lists - Adjust AutoFinMarketStep to calculate forecasts based on equal weights and direction signs - Change market.yaml instructions to require direction classification instead of similarity scoring - Modify tests to reflect direction-based classification and verify uniqueness across direction groups - Improve DingTalkWaitStep to support reconnect on server request with proper disconnect reason handling	2026-07-27 11:52:23 +08:00
jinliyl	1687179f84	feat: add Auto Fin cookbook and managed outbound proxy support (#392) ... * feat: add ssh proxy * feat: add ssh proxy * feat: add ssh proxy * feat: add ssh proxy * feat: add prompt * feat: add agent wrapper * feat: add agent wrapper * feat: add agent wrapper * feat: add tushare skill * feat: add tushare skill * feat: add tushare skill * feat: add none stream * chore(deps): update dependency versions in pyproject.toml - Bump claude-agent-sdk from 0.2.123 to 0.2.126 - Upgrade pre-commit to version 4.6.1 or higher - Upgrade pytest to version 9.1.1 or higher * feat(agent_wrapper): add session compaction support and unify session commands - Introduce compact_session method to BaseAgentWrapper and implement it in AsAgentWrapper, CcAgentWrapper, and CodexAgentWrapper - Add session_command module with SessionCommandResult dataclass and handle_session_command function for /clear and /compact commands - Update __init__.py exports to include session_command handlers - Modify DingTalkWaitStep to handle session commands via handle_session_command function - Remove streaming mode from DingTalkWaitStep and simplify reply handling to final Markdown replies only - Add unit tests for session compaction methods and session command handling across wrappers and DingTalk integration - Clean up and remove obsolete streaming and card rendering code from DingTalk wait step - Adjust daily_cookbook.yaml to remove stream and card_update_interval config entries for DingTalk wait step * feat(auto_fin): add Auto Fin simulated portfolio cookbook workflow - Add comprehensive Auto Fin schema exports for multiple models and enums - Implement base class and helpers for Auto Fin analysis steps - Create file, state, and formatting utilities for Auto Fin with atomic file writes and locking - Define Auto Fin pipeline with four analysis agents: backtest, event, portfolio, and US correlation - Register Auto Fin package in cookbook workflows and schema initialization - Add detailed documentation in markdown describing the system design, workflow, and data contracts * feat(outbound_proxy): add application-scoped outbound HTTP proxy components - Introduce BaseOutboundProxy and OutboundProxyEndpoint as core contracts - Implement FixedHttpOutboundProxy for external HTTP proxy integration - Add SshHttpOutboundProxy providing SSH-backed local HTTP proxy tunnels - Register outbound proxy components in component registry and enumeration - Update components package to include outbound_proxy module - Add dependency on pproxy for SSH HTTP proxy bridging - Include comprehensive unit tests covering proxy lifecycle, validation, environment merging, error handling, readiness, and monitoring mechanisms * refactor(network): replace SSH proxy with explicit HTTP outbound proxy - Remove SSH proxy helper implementation and references in codebase - Add support for explicit HTTP proxy URL in arXiv and HuggingFace clients - Modify clients to use async context manager for consistent resource handling - Update daily paper steps to forward outbound proxy configuration explicitly - Change tests to cover new proxy usage model and remove SSH proxy mocks - Add outbound proxy component configuration in daily_cookbook.yaml - Ensure proxy URL usage disables environment trust in HTTP clients - Fix app context component enum access to be defensive against missing keys * feat(agent_wrapper): add managed proxy support for command environments - Introduce BaseOutboundProxy binding in BaseAgentWrapper for outbound proxy management - Add bash_environment and command_proxy_environment properties to apply proxy settings - Update WorkspaceBackend instantiation in AsAgentWrapper to use bash_environment - Inject managed proxy export commands into Claude Code Bash commands via hooks - Enhance CodexAgentWrapper to include managed proxy in shell environment policy - Modify daily_cookbook.yaml steps to specify outbound_proxy as default where needed - Add comprehensive unit tests verifying managed proxy injection and environment isolation - Ensure subprocess_environment remains unchanged while proxy is applied selectively to commands * refactor(memory): replace search job_tools with memory in daily cookbook config - Change workspace_dir default from .reme to reme_workspace - Replace search job_tools with memory across multiple components and jobs - Update descriptions to reflect long-term memory retrieval instead of search - Modify system prompts to instruct using memory for retrieving notes - Adjust unit tests to verify memory job_tools and job presence instead of search - Ensure consistency in configuration and tests for memory backend usage * refactor(config): rename memory to memory_search in daily cookbook config - Change all occurrences of "memory" to "memory_search" in job_tools and job definitions - Update related system prompts to reflect the new memory_search terminology - Modify unit tests to assert the presence of memory_search instead of memory - Ensure consistency across skills, job tools, and backend configurations in multiple components * feat(auto_fin): add deterministic quantitative research and ranking fusion - Introduce new schema models: EtfScore, RankingMetrics, ExtremeAnalysis, DimensionRanking, and FusionRanking to represent deterministic research outputs - Add ranking data to event, backtest, us_correlation, and portfolio analysis outputs - Implement ranking_section renderer to format Top20 scores and diagnostics in Markdown - Develop AutoFinQuantStep for deterministic ETF ranking using TuShare data, Polars, and a custom extremely randomized tree ensemble - Integrate quantitative rankings into backtest and portfolio analysis steps and reports - Extend auto_fin pipeline with new quant_enabled and quant_required config options - Enforce ranking constraints like unique codes, contiguous ranks, and normalized fusion weights - Update analysis YAMLs with rules limiting data freshness, universe, and ranking usage - Incorporate ranking outputs into all major markdown report bodies in Auto Fin pipeline - Add concurrency-limited asynchronous TuShare client to fetch required market data - Introduce cross-sectional rank correlation and NDCG metrics for ranking quality evaluation * feat(auto_fin): implement stage-wise notification and reporting for analysis pipeline - Refactor notification config in daily_cookbook.yaml to support dispatch steps - Update AutoFinNotificationStep to deduplicate notifications per run stage - Add _notify_stage method in pipeline to send notifications for each analysis stage - Implement persistence and notification for event, backtest, US correlation, and portfolio stages - Modify pipeline flow to persist reports and notify after each stage completion - Adjust metadata to track notifications and errors per stage - Update tests to verify stage-wise notification sending and deduplication - Remove older combined report persistence in favor of modular stage handling * feat(auto_fin): add outbound proxy support for Tushare API usage - Introduce BaseOutboundProxy reference in AutoFinPipelineStep and AutoFinQuantStep - Update TushareResearchClient and trade calendar fetch to accept and use proxy URL - Create _ProxiedTushareApi adapter to route Tushare requests via explicit HTTP proxy - Modify create_tushare_api utility to optionally return proxied API client - Add unit tests covering proxy forwarding and client behavior with managed proxies - Ensure proxy usage respects explicit proxy URL over environment fallback - Integrate outbound proxy into data fetching and quantitative research steps * feat(auto_fin): enforce checkpoint time validation and add state models - Introduce AnalysisState base class and specific states for event, backtest, and US correlation analyses - Replace analysis output types with corresponding state classes in run schemas - Add require_checkpoint_reached method to validate decision_at/data_cutoff against current time - Enforce checkpoint time checks before analysis steps in event, backtest, portfolio, and quant analyses - Refactor quant data loading to include adjustment factors and apply price adjustments without fallback - Update analysis YAML docs to require real-time checkpoint validation and forbid using future data - Improve portfolio run serialization by excluding redundant legacy fields and nested proposed actions - Add helper to extract readable sections from persisted checkpoint documents - Fix event analysis output validation to reject events and sources with future timestamps * feat(auto_fin): auto-select latest reached checkpoint if none specified - Extend checkpoint config to accept empty string for auto selection - Add static method to compute latest checkpoint reached by current time - Modify pipeline step to auto-select checkpoint based on trade calendar and time - Adjust force flag default depending on whether checkpoint is explicit or auto - Log details when checkpoint is auto-selected to improve observability - Add comprehensive tests for auto checkpoint selection logic and edge cases - Remove deprecated default and required constraints from force parameter in config * refactor(auto_fin): unify datetime comparison with compare_datetimes utility - Replace direct datetime comparisons with compare_datetimes function calls - Use cmp_to_key with compare_datetimes for sorting datetime tuples and lists - Update validation logic in backtest, event, analysis, and ledger modules for consistent datetime handling - Add unit tests to verify handling of naive and aware datetime comparisons in event and backtest validations - Ensure marked_at and interval_end timestamps are set and compared consistently using compare_datetimes - Improve correctness of ordering and conditional checks related to timestamps throughout auto_fin steps and ledger code * feat(auto_fin): add datetime comparison helper for mixed timezone data - Implement compare_datetimes function to handle naive and aware datetimes - Ensure naive datetime is interpreted in the known timezone of the counterpart - Facilitate comparisons between legacy and timezone-aware Auto Fin data - Add module docstring explaining purpose of the helpers * docs(auto_fin): enforce unique ETF representative per sub-theme in analysis rules - Update backtest.yaml to recommend or highlight only one ETF per sub-theme for ETF analyses - Modify event.yaml to map only one representative ETF per sub-theme, avoiding duplicate recommendations - Revise portfolio.yaml to restrict holdings/buys to a single ETF per sub-theme, preventing repeated buys of highly overlapping ETFs - Adjust us_correlation.yaml to retain only one representative A-share ETF per sub-theme for mapping or recommendation - Add test to verify presence of new sub-theme uniqueness guidance in step prompts * feat(auto_fin): separate draft model and include deterministic fusion ranking - Introduce _PortfolioProposalDraft pydantic model for agent-authored fields before ranking - Discard any "fusion_ranking" data from draft to prevent conflicts with canonical ranking - Modify AutoFinPortfolioStep to receive draft, enrich with fusion_ranking, and produce final output - Update tests to use _PortfolioProposalDraft and validate deterministic fusion ranking propagation - Add async test verifying fusion ranking is correctly set in portfolio output with no errors * refactor(auto_fin): rewrite and simplify Auto Fin schema and steps - Remove legacy Auto Fin analysis step modules and helpers - Replace complex ranking and portfolio models with simplified current-news models - Update schema to focus on news-case workflow with new domain models - Remove A-share decision checkpoints and backtest details from schema - Simplify recommendation and decision output structures - Clean up deprecated state and utility functions - Update Auto Fin steps initialization to new pipeline steps only - Improve uniqueness validation for themes and ETFs in research plan * feat(auto_fin): implement full local cache and analysis workflow for Auto Fin - Add AutoFinDataStep to prepare and cache daily TuShare data with lookback - Add AutoFinAnalysisStep to analyze cached data and generate Markdown report - Implement detailed time window, ETF filtering, and historical case validation - Introduce YAML prompts for planning and decision-making steps - Update .gitignore to include reme_workspace/ - Clean up config and import structure for auto_fin steps - Remove old pipeline.py and consolidate functionality into new modules - Use polars for efficient CSV reading and data processing - Ensure atomic writes and strict JSON serialization for cache files - Enforce rules on news timing, ETF universe, and historical case usage * fix(auto_fin): restrict news data source to '财联社' in analysis and cache - Update analysis templates to specify current news as from '财联社' only - Modify news fetching functions to filter by source '财联社' - Add validation method to check cached news source correctness - Update news caching logic to exclude non-'财联社' news - Enhance unit tests with multiple sources to ensure filtering works - Confirm news API calls include source filter parameter as '财联社' * refactor(auto_fin): convert I/O methods to asynchronous implementations - Change _news, _dataset, and _theme_data methods to async for improved concurrency - Move JSONL and CSV reading operations to asynchronous wrappers using asyncio.to_thread - Remove synchronous _read_jsonl and _read_csv functions, integrate them as static async class methods - Update cache validation methods to async, awaiting I/O operations accordingly - Adjust usage of dataset and news retrieval in analysis step to await asynchronous methods - Add async unit test to validate JSONL reading with unicode line separators - Preserve existing functionality while enabling non-blocking file and data access * fix(nx_file_graph): defer networkx import and improve dependency handling - Move networkx import inside NxFileGraph constructor for lazy loading - Raise ImportError with original exception context if networkx is missing - Remove module-level fallback assignment of nx to None - Expand test to block loading of multiple optional core dependencies eagerly - Change exception type in test from ModuleNotFoundError to AssertionError - Update test comments to reflect broader optional dependency checks * feat(embedding_store): add quota retry delay mechanism for embedding requests - Introduce quota_retry_delay parameter to configure wait time before retry on quota exhaustion - Implement detection of insufficient quota errors in LocalEmbeddingStore without external SDK - Add retry logic with custom delay when quota is insufficient during embedding requests - Update configuration to set max_retries and quota_retry_delay defaults for embedding store - Add unit tests covering quota exhaustion retry behavior with delay and opt-in control - Ensure existing retry behavior remains unchanged if quota_retry_delay is not set * feat(auto_fin): add detailed logging to analysis and data fetching steps - Add _preview static method for bounded diagnostic output in analysis.py - Log prompt start, completion, errors, and validation details in _reply method - Add info logs for major processing steps in execute method of analysis.py - Add debug and info logs for cache validation, data fetching, and pagination in data.py - Log conditions for skipping reports and cache plans in data.py execute method - Log download summaries and cache writes for news and ETF data - Improve error logging with exception details in cache validation functions - Ensure all logs include context such as record counts, paths, and parameters * refactor(auto_fin): overhaul Auto Fin workflow and schema contracts - Replace old Auto Fin schema models with comprehensive new data classes - Remove legacy Auto Fin analysis step in favor of modular agent-based steps - Introduce AutoFinAgentStep for validating structured agent replies - Simplify data cleaning and JSONL writing utilities for news cache - Remove synchronous and asynchronous dataset methods from analysis step - Redefine Auto Fin analysis configuration for 360-day news retention and multi-step pipeline - Remove embedded analysis prompt templates and replace with agent-driven logic - Update __init__.py exports to match new step implementations and remove deprecated classes - Improve error handling and validation in agent step reply processing - Clean up redundant imports and unused code in analysis and data preparation modules * feat(auto_fin): add detailed logging for analysis and data processing steps - Add timing logs to measure agent prompt processing duration in analysis.py - Log news cache hits and news write paths with record counts in data.py - Include detailed info logs for news download start and completion in data.py - Add start, progress, and completion logs with topic and event counts in history.py - Log start and completion of merge step including path and ETF count in merge.py - Add start and done logs with window and news counts in topic.py * feat(auto_fin): enhance schema and steps with detailed ETF and event modeling - Replace and add multiple AutoFin schema classes to support detailed ETF selection, historical research, market analysis, forecast models, and report output with validation - Implement Shanghai timezone normalization and strict validation in schema models - Remove deprecated AutoFin analysis agent step and consolidate reply handling in base step - Introduce AutoFinStep base class with shared helpers for prompt handling, data fetching, logging, and JSONL file operations - Add AutoFinDataStep to manage daily news data complete with schedule validation, caching, and source validation logic - Update cookbook configuration to customize auto_fin step parameters and simplify outbound proxy settings - Refactor imports and clean unused code for better maintainability * feat(auto_fin): introduce detailed historical event resolution and market similarity analysis - Add AutoFinHistoricalEventReference and AutoFinHistoricalSimilarity models for refined event referencing and similarity judgment - Implement validation to ensure non-empty critical fields and uniqueness of historical news IDs - Develop method to resolve Agent-selected historical event references from workspace files with strict path and existence checks - Enrich historical events with market entry and future returns data after resolution - Redesign market step to calculate similarity-weighted ETF forecasts based on matched historical event similarities - Enforce validation on matched historical events for uniqueness and proper weight summation - Simplify merge step output to final Markdown report without YAML frontmatter and redundant fields - Update user instructions for history search, market, and merge steps to reflect new data structures and responsibilities - Adjust test suite to cover new schema and step behavior changes, including enhanced validation and JSON output formats * feat(auto_fin): add new cron jobs and output analysis jsonl - Add new cron jobs auto_fin_1145_cron and auto_fin_1800_cron with auto_fin_steps - Change auto_fin_0930_cron schedule to run Monday to Sunday - Extend merge step to write analysis data to auto_fin_analysis.jsonl - Update unit tests to verify new cron jobs and their steps configuration * fix(auto_fin): improve atomic file write and refresh daily index - Change temporary file naming to include UUID for uniqueness and hidden prefix - Replace atomic write method from using Path.replace to os.replace with safe unlink - Add import and use os.replace for safer file replace operation - Refresh daily index after writing auto finance markdown and JSONL files - Import and call refresh_day_index in merge step to update file index asynchronously * docs(cookbook): add optional SSH proxy configuration in README files - Introduce optional SSH proxy setup in auto-fin and daily_paper cookbooks - Provide instructions to enable outbound proxy via `daily_cookbook.yaml` and environment variables - Add `REME_PROXY_IP` and `REME_PROXY_ACCOUNT` environment variables descriptions in multiple README files - Update English and Chinese README and README_ZH documents with proxy details - Maintain consistent formatting of environment variable tables across documents * fix(file_io): include schema_version in hidden metadata keys - Added "schema_version" to _INDEX_HIDDEN_METADATA_KEYS in _daily_index.py - Updated _render_notes_block to always include additional keys regardless of schema_version fix(deps): move pproxy dependency to later in pyproject.toml - Removed pproxy from early dependencies list - Added pproxy back near the end of dependency list for better ordering fix(outbound_proxy): require pproxy package for ssh_http proxy - Added importlib.util check for pproxy package presence - Raise RuntimeError if pproxy is not installed when using SSH HTTP outbound proxy - Improved error message suggests installing reme-ai with 'core' extra * docs(readme): update News section with new Cookbook workflows - Clarify introduction of optional Cookbooks with Daily Paper and Auto Fin workflows - Update English README to reflect both paper discovery and file-native ETF event research - Revise Chinese README to include financial news and historical market data research capability - Maintain announcement of paper acceptance at Findings of ACL 2026 * feat(auto_fin): add calculation results to final Markdown output - Implement _calculation_results to summarize forecast for each ETF analyzed - Include program-calculated results in the JSON input for the Markdown report - Update YAML template to incorporate calculation results and adjust recommendation rules - Refine recommendation logic to rely on event impact judgments combined with calculation outputs - Modify tests to verify presence of calculation results and updated report content and format * up prompt * fix(keyword_index): ignore non-indexable chunks during keyword sync - Add is_indexable method to base and BM25 keyword index classes to check text tokenizability - Update local file store to exclude non-indexable chunks from expected document IDs to prevent rebuild - Fix JSONL chunker to correctly handle Unicode line separator U+2028 inside JSON strings without splitting - Add test to ensure non-empty but non-indexable chunk does not trigger keyword index rebuild - Add test to verify U+2028 character does not cause incorrect JSONL record splitting	2026-07-25 18:09:39 +08:00
jinliyl	46adb5ae1e	feat: add daily paper cookbook and DingTalk agent integration (#385) ... * feat(daily-paper): add daily paper cookbook workflow with schema and tests - Introduce daily paper schema types (DailyBriefOutput, PaperInfo, PaperNoteOutput, etc.) - Create daily paper cookbook module with analyze, collect, digest, rank, and select steps - Add cookbook entry point and integrate into main steps module - Replace job config export with daily brief output in schema exports - Add comprehensive unit tests covering pipeline, filtering, and output generation - Update dependencies including openai-codex and pypdf packages - Configure standalone daily paper cron job with proper scheduling and routing * test(daily_paper): update tests to use Claude Code wrapper exclusively - Add test to verify web search is disallowed by default in Claude Code - Update imports to include DailyBriefOutput, PaperNoteOutput, and PaperSelection schemas - Change test name from standalone_config_has_backend_split to reflect Claude Code only usage - Remove default agent wrapper and configure all steps to use Claude Code wrapper - Rename select_wrapper to cc_wrapper for clarity and consistency - Remove duplicate Claude Code wrapper initialization - Update test assertions to verify output schema usage matches expected sequence - Remove unused as_llm component from standalone configuration test * refactor(agent-wrapper): simplify skill resolution logic across all wrappers - Replace duplicate skill resolution code with centralized _resolve_project_skills method - Add project_path property with configurable relative path resolution - Introduce proper validation for skill names and directory existence - Change Codex wrapper to use project_path instead of workspace_path for skills - Add SKILL.md requirement validation for project skills - Remove redundant skill processing logic from individual wrappers * feat(daily_paper): add daily paper workflow with PDF analysis and brief generation - Implement shared state management and file helpers for daily-paper steps - Add PDF download and text extraction capabilities with arXiv integration - Create paper collection step with Hugging Face weekly/monthly rankings - Build ranking system using reciprocal-rank fusion with memory keyword scoring - Add Claude Code integration for paper analysis and detailed note generation - Implement digest step to create final five-minute brief from detailed notes - Add configuration for standalone daily cookbook application with cron scheduling - Create typed schema for paper information, selection, and output formats - Add atomic file writing with temporary file safety mechanisms - Implement exclusion logic for previously recommended papers and daily filters * feat(daily_paper): add DingTalk notification integration and enhance logging - Integrate DingTalk markdown send step to notify groups about daily paper briefs - Add comprehensive logging throughout daily paper workflow including start/finish events - Update daily paper analysis prompt to include code repository context requirement - Configure DingTalk notification in daily_cookbook.yaml with app credentials - Add dingtalk-stream dependency for proactive message API integration - Enhance daily paper README with DingTalk notification section and updated flow chart - Implement detailed logging for each step including paper processing and agent calls - Add test coverage for DingTalk markdown sending functionality and configuration - Update pre-commit config to exclude skills directory from checks - Add .claude/skills to gitignore for local development environment * refactor(dingtalk): move dingtalk_stream import to local scope and improve code safety - Moved global dingtalk_stream import to local scope in send.py to avoid eager loading - Added dynamic import with error handling for optional dependency cases - Updated test suite to verify lazy loading behavior works correctly - Fixed markdown title generation by using safe variable naming in wait.py - Enhanced test coverage for arxiv PDF download caching functionality - Updated application context initialization with proper resource directory configuration - Modified paper metadata to include source PDF path reference in output files * refactor(daily_paper): remove manifest system and store selection metadata in digest files - Remove JSON manifest creation and storage functionality - Store selection data directly in digest file frontmatter instead of separate manifest files - Add load_saved_selection method to rebuild selection from digest and paper-note metadata - Update README documentation to reflect new cookbook workflow architecture - Modify test cases to verify selection metadata in digest files instead of manifest JSON - Remove unused json import from multiple daily paper modules - Integrate PaperSelection schema for proper data validation in stored metadata * docs(daily_paper): add bilingual cookbook guides	2026-07-22 19:17:01 +08:00
jinliyl	7d86658f33	Refactor logging levels and add dream schema definitions (#291) ... * chore(logging): change info logs to debug level for data loading operations - Changed stopwords loading log from info to debug level - Changed file catalog nodes loading log from info to debug level - Changed file graph nodes loading log from info to debug level * feat(dream): add dream schema definitions and enum for auto-dream functionality - Add DreamBucketEnum with procedure, personal, and wiki values - Create comprehensive dream-related Pydantic models including DreamUnit, DreamTopic, DreamExtractOutput, IntegrateOutcome, TopicSelectionOutput, ProactiveResult, and DreamState - Move schema definitions from local step module to shared schema package - Update dream extraction and integration steps to use new enum-based bucket validation - Initialize digest directories for each dream bucket type - Enhance embedding store health check with workspace directory logging * refactor(tests): update DreamState import path in test_auto_dream.py - Move DreamState import from reme.steps.evolve.dream.schema to reme.schema - Maintain same functionality with updated module reference - Align import with new schema location in project structure	2026-06-24 16:44:03 +08:00
jinliyl	206a53e5ed	init: reme version 0.4.0 (#284)	2026-06-22 15:41:19 +08:00

Renamed from reme4/schema/__init__.py (Browse further)