154 commits

Author	SHA1	Message	Date
jinliyl	215c1f72f2	feat: refine local-first research and memory workflows (#444) ... * feat: refine local-first research workflows * fix: delegate structured output tool choice * refactor(auto-fin): fetch and filter rolling CLS news * fix(auto-fin): keep imports portable across platforms * feat(auto-fin): expose CLS fetch controls * fix(auto-fin): propagate configurable news window * feat(auto_fin): normalize hybrid wikilinks in report body - Add _normalize_hybrid_wikilinks method to remove redundant Markdown destinations - Use regex to identify hybrid wikilinks with optional destinations - Replace redundant destinations with simpler wikilink format for clarity - Ensure normalization is failure-safe with exception handling and logging - Update report body normalization process to apply hybrid wikilink fix - Add unit tests to verify correct normalization and failure safety behavior * fix(dream): serialize integration with application-wide asyncio lock - Add application-wide asyncio.Lock to serialize digest writes during integration - Update _snapshot_digest to capture metadata per bucket - Validate bucket association when recovering from file changes - Add tests ensuring recovery only from the correct bucket - Add tests confirming integration lock is shared across application context - Enhance strict topic YAML loading validation in dream utils - Add tests for strict topic loading rejecting invalid or lossy fields * fix(cookbook): enable configurable job_tools for digest and merge steps - Update daily_cookbook.yaml to add job_tools: [memory_search, read] in digest steps - Modify DailyPaperDigestStep to read job_tools from kwargs instead of fixed list - Modify AutoFinMergeStep to similarly read job_tools from kwargs - Update tests to pass job_tools explicitly when invoking these steps - Remove hardcoded _TOOLS constants and replace with dynamic job_tools handling * fix: retry incomplete dream receipts * perf(pdf): increase max PDF pages limit from 20 to 35 - Updated configuration max_pdf_pages from 20 to 35 in daily_cookbook.yaml - Modified code to extract up to 35 pages instead of 20 in analyze.py - Updated README and README_ZH to document the increased max_pdf_pages - Adjusted unit test assertions to reflect new max_pdf_pages limit of 35 * fix memory integration and daily paper links * docs clarify cookbook tool usage	2026-08-11 23:32:34 +08:00
jinliyl	9533c17d51	feat(web): serve workspace from HTTP service (#446) ... * feat(web): add the ReMe workspace frontend * feat(web): serve workspace from HTTP service * test(web): satisfy pylint docstring checks * fix(web): use same-origin API safely * fix(web): preserve API route semantics	2026-08-11 23:32:24 +08:00
imrewce	58276f740b	fix(file_io): auto appending suffix for all related steps (#430) ... * fix(file_io): auto appending suffix for all related steps * fix(file_io): covering boundary cases of potential directory input	2026-08-11 11:09:28 +08:00
jinliyl	21057931a9	fix(embedding): isolate caches by vector space (#442) ... * fix(embedding): isolate caches by vector space * fix(embedding): stabilize cache space switching * Revert "fix(embedding): stabilize cache space switching" This reverts commit 74193c9a0a. * fix(embedding): include resolved OpenAI endpoint in cache ID * fix(embedding): stabilize cache space switching * fix(embedding): isolate Ollama endpoint caches	2026-08-10 22:42:16 +08:00
jinliyl	072cb6a55b	feat(daily-paper): add opt-in Hugging Face mirror support (#437) ... * feat(daily-paper): add Hugging Face mirror switch * refactor(daily-paper): simplify the HF mirror switch and warn on ignored env The switch was a three-state bool|None where None preserved the legacy environment-driven selection, but no production caller ever passes None -- collect.py always resolves an explicit bool. Collapse it to a plain bool defaulting to False. HF_MIRROR_URL no longer redirects traffic on its own, so warn when it is configured while the mirror stays disabled; a mirror-only setup would otherwise fall back to the official site with no signal. Both READMEs now record the behavior change and stop presenting the two mirror variables as symmetric -- arXiv remains environment-driven while Hugging Face is gated on the job parameter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(daily-paper): address mirror configuration feedback --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>	2026-08-10 15:57:24 +08:00
jinliyl	d5e0d2837b	refactor: rebuild auto-fin and daily-paper cookbooks on structured-output agents (#432) ... * refactor: rebuild auto-fin and daily-paper cookbooks on structured-output agents Rework the auto-fin and daily-paper cookbooks to run on structured-output LLM agents instead of Claude Code agent wrappers, replace the SSH proxy with data-source mirrors, and rewrite the affected unit tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(auto_fin): unify JSON output serialization and writing - Extracted _write_output static method to serialize and write Pydantic models as compact JSON - Replaced inline JSON dump and write calls with _write_output usage across auto_fin steps - Added _report_path and _current_report for managing intra-day reports in AutoFinMergeStep - Updated auto_fin merge step to write output via new _write_output method - Enhanced news reading with caching in AutoFinHistoryStep - Refined returns calculation to handle events before close on non-trading days correctly feat(daily_paper): improve note path resolution and metadata handling - Introduced iter_note_metadata generator for safe Markdown frontmatter iteration - Added resolve_unique_note_path to avoid note filename conflicts on disk and in used titles - Updated analyze, collect, digest, and select steps to use centralized constants and helpers - Used utc_now_iso for consistent timestamping in metadata - Replaced direct frontmatter loads with iter_note_metadata in collect and analyze steps - Replaced hardcoded paper selection count with PAPER_COUNT constant in all relevant places - Added _MAX_SELECT_ATTEMPTS constant in select step for attempt management - Improved error messages for filename validation in daily paper title normalization feat(auto_fin): add multi-run cron schedules for intraday refinement - Defined three auto_fin cron jobs at 09:30, 11:30, and 18:00 Shanghai time for gradual report updates - Each intraday run adds evidence cumulatively instead of replacing prior output wholly - Updated daily_cookbook.yaml to register new cron schedules and remove legacy 12:00 cron refactor(auto_fin_data): clean ETF code handling and page limits - Replaced hardcoded DEFAULT_ETF_CODES with required non-empty config value "etf_codes" - Added constants for major news and fund page limits to control pagination - Improved ETF name extraction logic to handle missing fields consistently fix(auto_fin_merge): fix report retrieval and merging logic - Added support for getting current intra-day report in addition to previous day's report - Modified merge template to include prior and current report sections for better context - Adjusted report path handling to consistently use Path objects test(auto_fin): add coverage for returns calculation and report retrieval - Added test for returns when event occurs before close on non-trading day, checking next session entry - Added test for previous and current report retrieval feeding merge context with disk files - Extended test asserts for auto_fin cron schedule changes in config style(daily_paper): reorder and cleanup imports - Reorganized imports in _common.py for clarity and added missing collections.abc.Iterator import - Cleaned up commented and unused imports across daily_paper steps * feat: add configurable upstream mirror proxy * style: format auto-fin data step * fix: align cookbook mirrors and contracts --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-08-07 23:53:14 +08:00
jinliyl	e05b201da9	feat(backend): improve workspace support for web clients (#420) ... * feat(backend): improve workspace support for web clients * fix(config): preserve the default workspace directory * chore(reme): bump version to 0.4.1.5 - Update __version__ from 0.4.1.4 to 0.4.1.5 in initialization file * fix(chat): disable builtin tools in read-only mode * fix(agent): make builtin tools opt-in * fix(list): tolerate files removed during mtime sort * fix(chat): expose complete read-only job set	2026-08-07 23:52:56 +08:00
jinliyl	c5d92a24ab	feat: weave dream wikilinks into contextual prose (#428)	2026-08-06 17:40:01 +08:00
jinliyl	9218a2d0e3	refactor: derive dialog paths from session_dir (#421) ... * refactor: derive dialog paths from session directory * fix: normalize configured session paths * fix: align dialog watch paths with writers * fix: reject absolute session directories	2026-08-06 17:07:14 +08:00
Ziyang Guo	6503e1271c	fix(prompt): default omitted conditional flags to false (#424) ... Always apply conditional-line filtering so tagged prompt lines are removed unless the corresponding boolean flag is explicitly true. Add regressions for omitted flags with and without format variables. Test: pytest tests/unit/test_prompt_handler.py -q	2026-08-06 16:22:59 +08:00
jinliyl	f31daf1949	Revert "feat(backend): improve workspace support for web clients (#417)" (#419) ... This reverts commit b00eb0a9ea.	2026-08-05 23:17:08 +08:00
jinliyl	b00eb0a9ea	feat(backend): improve workspace support for web clients (#417)	2026-08-05 23:07:05 +08:00
lichen2015	ad4f23e4dc	feat(file_store): add ZvecLocalFileStore backend (#410) ... * feat(file_store): add ZvecLocalFileStore backend - Implement ZvecLocalFileStore with native zvec collection for ANN search. - Keep JSONL chunks as the source of truth; rebuild collection from chunks when sidecar digest/dimension/HNSW M mismatch is detected. - Add dedicated unit tests in tests/unit/test_zvec_file_store.py. - Parametrize existing file_store consistency tests to cover both LocalFileStore and ZvecLocalFileStore. - Register the new backend in reme/components/file_store/__init__.py. * fix(file_store): fix zvec collection sync and content validation, declare zvec dependency	2026-08-05 22:08:30 +08:00
xyf2020	5bc46c88b6	feat(benchmark): enhance session memory retrieval and isolate benchmark assets (#409) ... * chore(benchmark): isolate dataset/workspaces/results per benchmark - Move shared benchmark/{datasets,memory_workspaces,results} into per-benchmark subdirs benchmark/<name>/{dataset,workspaces,results} - Update beam/longmemeval config.yaml and run.py path defaults - Relocate longmemeval download.py to benchmark/longmemeval/ (downloads into dataset/ subdir); inline dataset download docs into README - Update .gitignore: benchmark/*/{dataset,workspaces,results}/ - Move result-{beam,longmemeval}.md to benchmark/results_md/ and drop result- prefix; update README links - Fix stale path refs in llm_judge.py and logs/demo_search_format.py * feat(benchmark): add read tool to agentic answer and update BEAM results - Add 'read' to job_tools in BaseAgenticAnswerStep for file reading capability - Document read tool usage in lme/agentic_answer.yaml system prompt - Update result-beam.md with latest evaluation scores (OVERALL: 0.623/0.580) * feat(auto_memory): add source line-number markers for note traceability - Add _format_history hook in AutoMemoryStep with line-number annotation - Override in BeamAutoMemoryStep to prefix each turn with [Ln] for citation - Add session_file variable to prompt templates for source marker paths - Simplify repeated extraction rules by referencing system prompt - Enhance agentic_answer search strategy (multi-search, read tool hint) - Add warning log on ReadStep failure * feat(beam): enhance auto_memory with source markers and pilot ingest tooling * refactor(beam): rename max_chunk_words to max_segment_words, drop one-off pilot scripts * feat: add CompressorStep and search_v2 dual-mode session compression - Add CompressorStep (reme/steps/evolve/compressor.py) for direct LLM text compression with optional query-guided relevance filtering - Extend search_v2_step to support query-aware and query-independent session transcript compression via _compress injected kwargs - Refactor _source_format.py: split into render_chunk_entries + join_chunk_entries; session chunks now render line-aligned with L<n>: prefixes for verbatim/compressed parity - Add JOB_TOOLS and INJECTED_JOB_KWARGS to BaseAgenticAnswerStep for per-subclass tool and parameter injection - LmeAgenticAnswerStep injects _search._compress payload to enable query-aware compression during benchmark evaluation - Record compression ablation results in result-longmemeval.md - Add unit tests for CompressorStep and search compression paths * refactor(compress): relax session compression to lenient format-preserving strategy and update LME results * refactor(benchmark): make session compression config-driven via compress_session flag Move session-transcript compression from LME hard-coded injection to a runtime context flag set by evaluation.compress_session in each benchmark config. Compression is off by default for both BEAM and LME, and BaseAgenticAnswerStep now conditionally injects the _search compress payload only when the flag is truthy. * feat(lme/auto_memory): add source attribution markers with line numbers Add _format_history to annotate each turn with [Ln] line numbers and expose {session_file} in prompts so the agent can emit bare wikilink-style source markers like [[session/dialog/s1.jsonl#L1-L2,L5-L6]] at the end of factual entries. Consolidate the per-prompt body/format rules into references to the system prompt to avoid drift, and add frontmatter- protection guidance for the edit tool. * feat: improve agentic answer prompt and update beam 100K results - Strengthen abstention rule: prohibit extrapolation from related but non-direct evidence - Add multi-angle search after preliminary answer to check for conflicting/supplementary/updated information - Add max-iteration fallback to 'Information not found' - Update beam.md with 100K results (agentscope 2.0.4.post1, from scratch) including per-type token consumption and memory construction stats - config.yaml: 100K dataset, 20 workers for BEAM evaluation - run.py: add memory construction token usage tracking (default agent) - Overall: 0.635 → 0.654 (+0.019), contradiction_resolution: 0.338 → 0.478 (+0.140), abstention: 0.500 → 0.525 (+0.025) * feat(read): add session-aware formatting for read tool and update BEAM eval - Add truncate_session_output in _file_io.py to render jsonl session lines as [speaker @ time] content before byte-budget truncation - Add read_step_format_session flag to ReadStep, honoring injected job kwargs (precedence) and YAML fallback - Inject read_step_format_session=True into BaseAgenticAnswerStep so agentic answer reads render session transcripts human-readably - Refine BEAM agentic_answer prompt: continue multi-angle search after preliminary answer, forbid fabrication/extrapolation - Update BEAM config to 1M variant and add sequential 100K-eval / 1M-build shell script - Refresh benchmark/results_md/beam.md with latest results * chore(config): disable expand_links in beam and lme search_v2 configs * refactor(beam): drop one-off sequential 100K-eval-then-1M-build script * fix(benchmark): add compressor job to beam config and fix BEAM clone instructions - Add compressor job and compressor as_llm component to reme/config/beam.yaml (aligned with lme.yaml) so that compress_session: true works for BEAM - Add graceful degradation guard in search_v2._compress_session_entries: when the compressor job is missing from the active config, log a warning and skip compression instead of raising 'Job compressor not found'. Skipped when there is no app_context so unit tests mocking run_job still drive compression behavior. - Fix BEAM download instructions in README.md/README_ZH.md: add mkdir -p before cd benchmark/beam/dataset (the directory is gitignored and absent in a fresh clone) * fix(steps): guard compressor exceptions and fix ReadStep boolean override 1. search_v2: catch per-entry exceptions from run_job('compressor') inside compress() so asyncio.gather never propagates a compressor failure (e.g. temporary LLM outage). The failing entry keeps its original body while remaining entries are still compressed, preserving already-retrieved search results. 2. read: replace 'context_value or yaml_value' with an existence check so that a runtime-injected False can explicitly disable a YAML-true read_step_format_session flag. Add focused unit tests for both paths. * fix(search_v2): use existence check for strict_date_filter boolean override Replace 'context_value or yaml_value' with an existence-based check so that a runtime-injected False can explicitly disable a YAML-true strict_date_filter flag, consistent with the read_step_format_session fix. * refactor(search): simplify strict_date_filter fallback to truthiness-or * style(test): rename unused param to satisfy pylint W0613 --------- Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>	2026-08-05 19:23:42 +08:00
jinliyl	e256c556ca	feat: add workspace web APIs and star growth report (#416)	2026-08-05 18:03:37 +08:00
jinliyl	eac8223387	feat: add frontend-ready wikilink graph APIs (#414)	2026-08-05 16:45:50 +08:00
jinliyl	a9ec334adc	feat: simplify wikilink semantics and support line anchors (#412) ... * feat: simplify local links and support line anchors * fix: align line anchor tests with CI lint * fix: preserve local links across file moves * fix: encode markdown paths when rewriting links * refactor(read): keep explicit line range parameters * fix: simplify legacy link predicate compatibility * docs: align local link behavior with implementation * fix: skip unsupported markdown destination escapes * fix: normalize workspace link paths across platforms * fix: bound markdown link scanning * fix: keep local link processing linear * docs: clarify permissive markdown link parsing * fix: handle local link processing failures * refactor: limit file links to wikilink syntax * docs: align wikilink contract with implementation * fix: normalize dream and neighbor paths on Windows * fix: resolve workspace path for neighbor expansion	2026-08-05 11:47:50 +08:00
xyf2020	6b035c6553	feat(evaluation): track job calls and agent token usage in benchmarks (#406) ... * feat(counter): extend counter tree utils and record job call statistics - replace global_counter_next with fetch-and-add style global_counter_add/inc, plus read-only global_counter_get and global_counter_get_all - record per-job call counts in app_context.metadata via BaseJob._record_call, covering background/cron/stream jobs - update agentic_answer step and utils exports; add unit tests for job counting and counter utils * feat(evaluation): add check_job_count interface and report search calls in benchmarks - Extract _counter_key from BaseJob._record_call for reusable counter lookup - Add reme.utils.evaluation_interface.check_job_count read-only helper - Track and report average search calls per query in beam and longmemeval benchmarks * job counter * token消耗量统计 * benchmark输出完整token消耗统计 * benchmark统计输出改用标准差 - beam/longmemeval 的工具调用与 token 统计由方差改为标准差输出 - 修复 lint: 局部变量遮蔽 importlib.metadata、补充测试 docstring - black 格式化 * fix(evaluation): preserve complete token usage metrics * fix: exclude stream replies from token accounting * Revert "fix: exclude stream replies from token accounting" This reverts commit 85bf32064d. * Reapply "fix: exclude stream replies from token accounting" This reverts commit 6722c24dc5. * support agent scope 2.0.5 * feat: support injection_config to disable runtime state injection in benchmarks - Add InjectionConfig passthrough in AsAgentWrapper.reply() - Disable inject_runtime_state in BaseAgenticAnswerStep to avoid wall-clock time conflicting with benchmark query_time anchors - Disable inject_runtime_state in beam/lme llm_judge calls * feat: agentscope dual-version compat & benchmark improvements - Add version_tuple utility for semantic version comparison - AsAgentWrapper: version-aware InjectionConfig, max_iters doubling, and token usage collection (reply vs reply_stream) for AS>=2.0.5/<2.0.5 - Default inject_runtime_state=False in wrapper to avoid benchmark time-anchor conflicts; remove per-callsite injection_config overrides - longmemeval run.py: support question_ids filter in dataset config - Fix unused import in test_evaluation_interface; format fixes * chore: remove temporary flip-test benchmark config * revert: pin agentscope to 2.0.4.post1 and drop dual-version compat * fix(evaluation): clarify usage semantics and atomic counters --------- Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135> Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>	2026-08-04 11:42:18 +08:00
Sen Huang	550317c3bf	Revert "feat(plugin): add ReMe integration for Codex (#372)" (#400) ... This reverts commit a367c2ce13.	2026-07-29 18:14:05 +08:00
DiegoCluv7	a367c2ce13	feat(plugin): add ReMe integration for Codex (#372) ... * feat(plugin): add ReMe integration for Codex * fix(plugin): fix Codex plugin port, transcript ingestion, and Windows support * fix(plugin): correct Codex transcript schema, path validation, and hook fixes * test(plugin): add MCP round-trip tests * fix(plugin): rewrite parser and tests. * fix(plugin): reserve id-less messages, cover marketplace manifest, error handling, path fixes, and main sync Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-07-29 18:11:29 +08:00
jinliyl	c937be9d94	refactor(auto_fin): normalize data models and selection logic across agents (#396) ... - Introduce tolerant AutoFinAgentModel base class allowing extra fields in raw Agent outputs - Replace strict models with tolerant ones for ETF, historical event, market selection, and report outputs - Remove redundant field validators and allow empty defaults for key string fields - Enhance historical source path resolution to safely filter invalid or out-of-workspace paths - Add normalization of whitespace and validation to historical event references before processing - Implement normalization in Topic and Market Agent selections to eliminate duplicates, blanks, unknowns - Limit Topic Agent output to top 20 ETFs and ensure sorting and deduplication of events - Normalize final Markdown report by removing redundant headers and providing safe fallbacks - Update agent prompts to clarify task constraints and improve instruction consistency - Add extensive tests for normalization, filtering, and safe source resolution for historical events	2026-07-27 20:15:27 +08:00
xyf2020	4eb2adf961	feat(faiss_file_store): upgrade FAISS to HNSW index with async reindex (#390) ... * feat(file_store): upgrade FAISS to HNSW index with async reindex and path constraint - Replace IndexFlatIP with IndexHNSWFlat for better recall/speed tradeoff - Add dynamic efSearch (limit * 5) scaled to query request size - Add async_reindex option: background rebuild with generation-based invalidation - Extract _delete_nodes() in LocalFileStore for subclass reuse - Add unit tests for file store consistency * fix: resolve pylint warnings in faiss store and test file * refactor(file_store): replace generation-based reindex with event-flag worker - Replace _reindex_generation/lock/task with a single long-lived worker coroutine consuming an asyncio.Event flag; repeated submissions coalesce - Use local index reference in vector_search to avoid TOCTOU on self._faiss_index - Pass index explicitly to _set_ef_search for consistency - Track _index_writes to re-arm reindex after concurrent writes - Update tests to match new internal API * fix: resolve pylint too-many-return-statements and implicit-booleaness warnings * feat(file_store): add refine maintenance hook and incremental embedding backfill - Add refine() idle-time maintenance hook to BaseFileStore/LocalFileStore - FaissLocalFileStore: incremental vector add on backfill instead of full rebuild - Dynamic tombstone compaction threshold scaled by index size - Add RefineStoreStep with daily cron job (refine_store_cron) - Enable faiss backend and embedding_store by default in default.yaml - Add unit tests for faiss index maintenance * chore(deps): promote faiss-cpu to core dependencies faiss backend is now the default file_store, so faiss-cpu moves from the optional [core] extra to the base dependencies list. * feat: rename refine_store to optimize_index and add vecdb_path_constraint - Rename refine_store step to optimize_index with cron job scheduling - Add vecdb_path_constraint to file_store components - Update default.yaml with optimize_index_cron and faiss backend comment - Update memory_search docs (en/zh) for FAISS vector management - Update unit tests for index maintenance * feat(faiss): add embedding digest to reject stale sidecar after partial dump Add _chunks_embedding_digest() that computes an order-independent SHA-256 over (chunk_id, float16 embedding) pairs. The digest is written into the idmap sidecar at dump time and verified at load time. A mismatch means the sidecar vectors belong to a different chunk generation than the authoritative JSONL — detectable even when the live-ID set is unchanged (same-ID in-place update crash window). Add test_faiss_rejects_stale_sidecar_after_partial_dump reproducing the crash-between-writes scenario and asserting digest-based rejection. Compress verbose docstrings/comments in existing tests for pylint line budget. --------- Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>	2026-07-27 19:54:38 +08:00
xyf2020	f34dcdb09b	feat(Step tools): add white/black path prefix permission filtering to read, edit, write (#391) ... * feat(read): add white/black path prefix permission filtering to ReadStep * feat: add PrefixCheck mixin for path-prefix permission in file I/O steps * feat: add injected_job_kwargs mechanism and refine path-prefix permission * refactor(file_io): consolidate prefix_check into _path module	2026-07-27 17:20:21 +08:00
jinliyl	2f79977df0	refactor(auto_fin): replace similarity with direction classification for historical events (#395) ... - Add AutoFinHistoricalDirectionReference model to classify historical events by direction - Remove AutoFinHistoricalSimilarity and related similarity score usage - Update AutoFinMarketSelection to handle same and opposite direction event lists - Adjust AutoFinMarketStep to calculate forecasts based on equal weights and direction signs - Change market.yaml instructions to require direction classification instead of similarity scoring - Modify tests to reflect direction-based classification and verify uniqueness across direction groups - Improve DingTalkWaitStep to support reconnect on server request with proper disconnect reason handling	2026-07-27 11:52:23 +08:00
Amir Fathi	0522135791	fix(file_io): stop ReadStep small-file path over-counting total lines by 1 (#389) ... content.split("\n") yields a trailing empty element for any file ending in a newline, inflating total by 1 and letting a start_line one past real EOF be silently accepted instead of rejected. Mirrors the trailing-newline correction default_file_chunker already applies, and matches the large-file path's line-by-line count. Fixes #388	2026-07-27 11:01:09 +08:00
jinliyl	11fe50d89c	refactor(auto_fin/history_search): improve historical event resolution and error handling (#394) ... - Separate candidate source file resolution from event resolution logic - Allow fallback to date-derived daily news file if original source is missing - Check existence and validity of historical source files more robustly - Handle multiple candidate source files and aggregate matches before validation - Gather and log resolution limitations without stopping processing - Return resolved events along with a list of resolution warnings - Update related code to consume new return signature and merge limitations - Add detailed validation on source path relativity and file naming conventions	2026-07-26 19:00:46 +08:00
jinliyl	1687179f84	feat: add Auto Fin cookbook and managed outbound proxy support (#392) ... * feat: add ssh proxy * feat: add ssh proxy * feat: add ssh proxy * feat: add ssh proxy * feat: add prompt * feat: add agent wrapper * feat: add agent wrapper * feat: add agent wrapper * feat: add tushare skill * feat: add tushare skill * feat: add tushare skill * feat: add none stream * chore(deps): update dependency versions in pyproject.toml - Bump claude-agent-sdk from 0.2.123 to 0.2.126 - Upgrade pre-commit to version 4.6.1 or higher - Upgrade pytest to version 9.1.1 or higher * feat(agent_wrapper): add session compaction support and unify session commands - Introduce compact_session method to BaseAgentWrapper and implement it in AsAgentWrapper, CcAgentWrapper, and CodexAgentWrapper - Add session_command module with SessionCommandResult dataclass and handle_session_command function for /clear and /compact commands - Update __init__.py exports to include session_command handlers - Modify DingTalkWaitStep to handle session commands via handle_session_command function - Remove streaming mode from DingTalkWaitStep and simplify reply handling to final Markdown replies only - Add unit tests for session compaction methods and session command handling across wrappers and DingTalk integration - Clean up and remove obsolete streaming and card rendering code from DingTalk wait step - Adjust daily_cookbook.yaml to remove stream and card_update_interval config entries for DingTalk wait step * feat(auto_fin): add Auto Fin simulated portfolio cookbook workflow - Add comprehensive Auto Fin schema exports for multiple models and enums - Implement base class and helpers for Auto Fin analysis steps - Create file, state, and formatting utilities for Auto Fin with atomic file writes and locking - Define Auto Fin pipeline with four analysis agents: backtest, event, portfolio, and US correlation - Register Auto Fin package in cookbook workflows and schema initialization - Add detailed documentation in markdown describing the system design, workflow, and data contracts * feat(outbound_proxy): add application-scoped outbound HTTP proxy components - Introduce BaseOutboundProxy and OutboundProxyEndpoint as core contracts - Implement FixedHttpOutboundProxy for external HTTP proxy integration - Add SshHttpOutboundProxy providing SSH-backed local HTTP proxy tunnels - Register outbound proxy components in component registry and enumeration - Update components package to include outbound_proxy module - Add dependency on pproxy for SSH HTTP proxy bridging - Include comprehensive unit tests covering proxy lifecycle, validation, environment merging, error handling, readiness, and monitoring mechanisms * refactor(network): replace SSH proxy with explicit HTTP outbound proxy - Remove SSH proxy helper implementation and references in codebase - Add support for explicit HTTP proxy URL in arXiv and HuggingFace clients - Modify clients to use async context manager for consistent resource handling - Update daily paper steps to forward outbound proxy configuration explicitly - Change tests to cover new proxy usage model and remove SSH proxy mocks - Add outbound proxy component configuration in daily_cookbook.yaml - Ensure proxy URL usage disables environment trust in HTTP clients - Fix app context component enum access to be defensive against missing keys * feat(agent_wrapper): add managed proxy support for command environments - Introduce BaseOutboundProxy binding in BaseAgentWrapper for outbound proxy management - Add bash_environment and command_proxy_environment properties to apply proxy settings - Update WorkspaceBackend instantiation in AsAgentWrapper to use bash_environment - Inject managed proxy export commands into Claude Code Bash commands via hooks - Enhance CodexAgentWrapper to include managed proxy in shell environment policy - Modify daily_cookbook.yaml steps to specify outbound_proxy as default where needed - Add comprehensive unit tests verifying managed proxy injection and environment isolation - Ensure subprocess_environment remains unchanged while proxy is applied selectively to commands * refactor(memory): replace search job_tools with memory in daily cookbook config - Change workspace_dir default from .reme to reme_workspace - Replace search job_tools with memory across multiple components and jobs - Update descriptions to reflect long-term memory retrieval instead of search - Modify system prompts to instruct using memory for retrieving notes - Adjust unit tests to verify memory job_tools and job presence instead of search - Ensure consistency in configuration and tests for memory backend usage * refactor(config): rename memory to memory_search in daily cookbook config - Change all occurrences of "memory" to "memory_search" in job_tools and job definitions - Update related system prompts to reflect the new memory_search terminology - Modify unit tests to assert the presence of memory_search instead of memory - Ensure consistency across skills, job tools, and backend configurations in multiple components * feat(auto_fin): add deterministic quantitative research and ranking fusion - Introduce new schema models: EtfScore, RankingMetrics, ExtremeAnalysis, DimensionRanking, and FusionRanking to represent deterministic research outputs - Add ranking data to event, backtest, us_correlation, and portfolio analysis outputs - Implement ranking_section renderer to format Top20 scores and diagnostics in Markdown - Develop AutoFinQuantStep for deterministic ETF ranking using TuShare data, Polars, and a custom extremely randomized tree ensemble - Integrate quantitative rankings into backtest and portfolio analysis steps and reports - Extend auto_fin pipeline with new quant_enabled and quant_required config options - Enforce ranking constraints like unique codes, contiguous ranks, and normalized fusion weights - Update analysis YAMLs with rules limiting data freshness, universe, and ranking usage - Incorporate ranking outputs into all major markdown report bodies in Auto Fin pipeline - Add concurrency-limited asynchronous TuShare client to fetch required market data - Introduce cross-sectional rank correlation and NDCG metrics for ranking quality evaluation * feat(auto_fin): implement stage-wise notification and reporting for analysis pipeline - Refactor notification config in daily_cookbook.yaml to support dispatch steps - Update AutoFinNotificationStep to deduplicate notifications per run stage - Add _notify_stage method in pipeline to send notifications for each analysis stage - Implement persistence and notification for event, backtest, US correlation, and portfolio stages - Modify pipeline flow to persist reports and notify after each stage completion - Adjust metadata to track notifications and errors per stage - Update tests to verify stage-wise notification sending and deduplication - Remove older combined report persistence in favor of modular stage handling * feat(auto_fin): add outbound proxy support for Tushare API usage - Introduce BaseOutboundProxy reference in AutoFinPipelineStep and AutoFinQuantStep - Update TushareResearchClient and trade calendar fetch to accept and use proxy URL - Create _ProxiedTushareApi adapter to route Tushare requests via explicit HTTP proxy - Modify create_tushare_api utility to optionally return proxied API client - Add unit tests covering proxy forwarding and client behavior with managed proxies - Ensure proxy usage respects explicit proxy URL over environment fallback - Integrate outbound proxy into data fetching and quantitative research steps * feat(auto_fin): enforce checkpoint time validation and add state models - Introduce AnalysisState base class and specific states for event, backtest, and US correlation analyses - Replace analysis output types with corresponding state classes in run schemas - Add require_checkpoint_reached method to validate decision_at/data_cutoff against current time - Enforce checkpoint time checks before analysis steps in event, backtest, portfolio, and quant analyses - Refactor quant data loading to include adjustment factors and apply price adjustments without fallback - Update analysis YAML docs to require real-time checkpoint validation and forbid using future data - Improve portfolio run serialization by excluding redundant legacy fields and nested proposed actions - Add helper to extract readable sections from persisted checkpoint documents - Fix event analysis output validation to reject events and sources with future timestamps * feat(auto_fin): auto-select latest reached checkpoint if none specified - Extend checkpoint config to accept empty string for auto selection - Add static method to compute latest checkpoint reached by current time - Modify pipeline step to auto-select checkpoint based on trade calendar and time - Adjust force flag default depending on whether checkpoint is explicit or auto - Log details when checkpoint is auto-selected to improve observability - Add comprehensive tests for auto checkpoint selection logic and edge cases - Remove deprecated default and required constraints from force parameter in config * refactor(auto_fin): unify datetime comparison with compare_datetimes utility - Replace direct datetime comparisons with compare_datetimes function calls - Use cmp_to_key with compare_datetimes for sorting datetime tuples and lists - Update validation logic in backtest, event, analysis, and ledger modules for consistent datetime handling - Add unit tests to verify handling of naive and aware datetime comparisons in event and backtest validations - Ensure marked_at and interval_end timestamps are set and compared consistently using compare_datetimes - Improve correctness of ordering and conditional checks related to timestamps throughout auto_fin steps and ledger code * feat(auto_fin): add datetime comparison helper for mixed timezone data - Implement compare_datetimes function to handle naive and aware datetimes - Ensure naive datetime is interpreted in the known timezone of the counterpart - Facilitate comparisons between legacy and timezone-aware Auto Fin data - Add module docstring explaining purpose of the helpers * docs(auto_fin): enforce unique ETF representative per sub-theme in analysis rules - Update backtest.yaml to recommend or highlight only one ETF per sub-theme for ETF analyses - Modify event.yaml to map only one representative ETF per sub-theme, avoiding duplicate recommendations - Revise portfolio.yaml to restrict holdings/buys to a single ETF per sub-theme, preventing repeated buys of highly overlapping ETFs - Adjust us_correlation.yaml to retain only one representative A-share ETF per sub-theme for mapping or recommendation - Add test to verify presence of new sub-theme uniqueness guidance in step prompts * feat(auto_fin): separate draft model and include deterministic fusion ranking - Introduce _PortfolioProposalDraft pydantic model for agent-authored fields before ranking - Discard any "fusion_ranking" data from draft to prevent conflicts with canonical ranking - Modify AutoFinPortfolioStep to receive draft, enrich with fusion_ranking, and produce final output - Update tests to use _PortfolioProposalDraft and validate deterministic fusion ranking propagation - Add async test verifying fusion ranking is correctly set in portfolio output with no errors * refactor(auto_fin): rewrite and simplify Auto Fin schema and steps - Remove legacy Auto Fin analysis step modules and helpers - Replace complex ranking and portfolio models with simplified current-news models - Update schema to focus on news-case workflow with new domain models - Remove A-share decision checkpoints and backtest details from schema - Simplify recommendation and decision output structures - Clean up deprecated state and utility functions - Update Auto Fin steps initialization to new pipeline steps only - Improve uniqueness validation for themes and ETFs in research plan * feat(auto_fin): implement full local cache and analysis workflow for Auto Fin - Add AutoFinDataStep to prepare and cache daily TuShare data with lookback - Add AutoFinAnalysisStep to analyze cached data and generate Markdown report - Implement detailed time window, ETF filtering, and historical case validation - Introduce YAML prompts for planning and decision-making steps - Update .gitignore to include reme_workspace/ - Clean up config and import structure for auto_fin steps - Remove old pipeline.py and consolidate functionality into new modules - Use polars for efficient CSV reading and data processing - Ensure atomic writes and strict JSON serialization for cache files - Enforce rules on news timing, ETF universe, and historical case usage * fix(auto_fin): restrict news data source to '财联社' in analysis and cache - Update analysis templates to specify current news as from '财联社' only - Modify news fetching functions to filter by source '财联社' - Add validation method to check cached news source correctness - Update news caching logic to exclude non-'财联社' news - Enhance unit tests with multiple sources to ensure filtering works - Confirm news API calls include source filter parameter as '财联社' * refactor(auto_fin): convert I/O methods to asynchronous implementations - Change _news, _dataset, and _theme_data methods to async for improved concurrency - Move JSONL and CSV reading operations to asynchronous wrappers using asyncio.to_thread - Remove synchronous _read_jsonl and _read_csv functions, integrate them as static async class methods - Update cache validation methods to async, awaiting I/O operations accordingly - Adjust usage of dataset and news retrieval in analysis step to await asynchronous methods - Add async unit test to validate JSONL reading with unicode line separators - Preserve existing functionality while enabling non-blocking file and data access * fix(nx_file_graph): defer networkx import and improve dependency handling - Move networkx import inside NxFileGraph constructor for lazy loading - Raise ImportError with original exception context if networkx is missing - Remove module-level fallback assignment of nx to None - Expand test to block loading of multiple optional core dependencies eagerly - Change exception type in test from ModuleNotFoundError to AssertionError - Update test comments to reflect broader optional dependency checks * feat(embedding_store): add quota retry delay mechanism for embedding requests - Introduce quota_retry_delay parameter to configure wait time before retry on quota exhaustion - Implement detection of insufficient quota errors in LocalEmbeddingStore without external SDK - Add retry logic with custom delay when quota is insufficient during embedding requests - Update configuration to set max_retries and quota_retry_delay defaults for embedding store - Add unit tests covering quota exhaustion retry behavior with delay and opt-in control - Ensure existing retry behavior remains unchanged if quota_retry_delay is not set * feat(auto_fin): add detailed logging to analysis and data fetching steps - Add _preview static method for bounded diagnostic output in analysis.py - Log prompt start, completion, errors, and validation details in _reply method - Add info logs for major processing steps in execute method of analysis.py - Add debug and info logs for cache validation, data fetching, and pagination in data.py - Log conditions for skipping reports and cache plans in data.py execute method - Log download summaries and cache writes for news and ETF data - Improve error logging with exception details in cache validation functions - Ensure all logs include context such as record counts, paths, and parameters * refactor(auto_fin): overhaul Auto Fin workflow and schema contracts - Replace old Auto Fin schema models with comprehensive new data classes - Remove legacy Auto Fin analysis step in favor of modular agent-based steps - Introduce AutoFinAgentStep for validating structured agent replies - Simplify data cleaning and JSONL writing utilities for news cache - Remove synchronous and asynchronous dataset methods from analysis step - Redefine Auto Fin analysis configuration for 360-day news retention and multi-step pipeline - Remove embedded analysis prompt templates and replace with agent-driven logic - Update __init__.py exports to match new step implementations and remove deprecated classes - Improve error handling and validation in agent step reply processing - Clean up redundant imports and unused code in analysis and data preparation modules * feat(auto_fin): add detailed logging for analysis and data processing steps - Add timing logs to measure agent prompt processing duration in analysis.py - Log news cache hits and news write paths with record counts in data.py - Include detailed info logs for news download start and completion in data.py - Add start, progress, and completion logs with topic and event counts in history.py - Log start and completion of merge step including path and ETF count in merge.py - Add start and done logs with window and news counts in topic.py * feat(auto_fin): enhance schema and steps with detailed ETF and event modeling - Replace and add multiple AutoFin schema classes to support detailed ETF selection, historical research, market analysis, forecast models, and report output with validation - Implement Shanghai timezone normalization and strict validation in schema models - Remove deprecated AutoFin analysis agent step and consolidate reply handling in base step - Introduce AutoFinStep base class with shared helpers for prompt handling, data fetching, logging, and JSONL file operations - Add AutoFinDataStep to manage daily news data complete with schedule validation, caching, and source validation logic - Update cookbook configuration to customize auto_fin step parameters and simplify outbound proxy settings - Refactor imports and clean unused code for better maintainability * feat(auto_fin): introduce detailed historical event resolution and market similarity analysis - Add AutoFinHistoricalEventReference and AutoFinHistoricalSimilarity models for refined event referencing and similarity judgment - Implement validation to ensure non-empty critical fields and uniqueness of historical news IDs - Develop method to resolve Agent-selected historical event references from workspace files with strict path and existence checks - Enrich historical events with market entry and future returns data after resolution - Redesign market step to calculate similarity-weighted ETF forecasts based on matched historical event similarities - Enforce validation on matched historical events for uniqueness and proper weight summation - Simplify merge step output to final Markdown report without YAML frontmatter and redundant fields - Update user instructions for history search, market, and merge steps to reflect new data structures and responsibilities - Adjust test suite to cover new schema and step behavior changes, including enhanced validation and JSON output formats * feat(auto_fin): add new cron jobs and output analysis jsonl - Add new cron jobs auto_fin_1145_cron and auto_fin_1800_cron with auto_fin_steps - Change auto_fin_0930_cron schedule to run Monday to Sunday - Extend merge step to write analysis data to auto_fin_analysis.jsonl - Update unit tests to verify new cron jobs and their steps configuration * fix(auto_fin): improve atomic file write and refresh daily index - Change temporary file naming to include UUID for uniqueness and hidden prefix - Replace atomic write method from using Path.replace to os.replace with safe unlink - Add import and use os.replace for safer file replace operation - Refresh daily index after writing auto finance markdown and JSONL files - Import and call refresh_day_index in merge step to update file index asynchronously * docs(cookbook): add optional SSH proxy configuration in README files - Introduce optional SSH proxy setup in auto-fin and daily_paper cookbooks - Provide instructions to enable outbound proxy via `daily_cookbook.yaml` and environment variables - Add `REME_PROXY_IP` and `REME_PROXY_ACCOUNT` environment variables descriptions in multiple README files - Update English and Chinese README and README_ZH documents with proxy details - Maintain consistent formatting of environment variable tables across documents * fix(file_io): include schema_version in hidden metadata keys - Added "schema_version" to _INDEX_HIDDEN_METADATA_KEYS in _daily_index.py - Updated _render_notes_block to always include additional keys regardless of schema_version fix(deps): move pproxy dependency to later in pyproject.toml - Removed pproxy from early dependencies list - Added pproxy back near the end of dependency list for better ordering fix(outbound_proxy): require pproxy package for ssh_http proxy - Added importlib.util check for pproxy package presence - Raise RuntimeError if pproxy is not installed when using SSH HTTP outbound proxy - Improved error message suggests installing reme-ai with 'core' extra * docs(readme): update News section with new Cookbook workflows - Clarify introduction of optional Cookbooks with Daily Paper and Auto Fin workflows - Update English README to reflect both paper discovery and file-native ETF event research - Revise Chinese README to include financial news and historical market data research capability - Maintain announcement of paper acceptance at Findings of ACL 2026 * feat(auto_fin): add calculation results to final Markdown output - Implement _calculation_results to summarize forecast for each ETF analyzed - Include program-calculated results in the JSON input for the Markdown report - Update YAML template to incorporate calculation results and adjust recommendation rules - Refine recommendation logic to rely on event impact judgments combined with calculation outputs - Modify tests to verify presence of calculation results and updated report content and format * up prompt * fix(keyword_index): ignore non-indexable chunks during keyword sync - Add is_indexable method to base and BM25 keyword index classes to check text tokenizability - Update local file store to exclude non-indexable chunks from expected document IDs to prevent rebuild - Fix JSONL chunker to correctly handle Unicode line separator U+2028 inside JSON strings without splitting - Add test to ensure non-empty but non-indexable chunk does not trigger keyword index rebuild - Add test to verify U+2028 character does not cause incorrect JSONL record splitting	2026-07-25 18:09:39 +08:00
jinliyl	46adb5ae1e	feat: add daily paper cookbook and DingTalk agent integration (#385) ... * feat(daily-paper): add daily paper cookbook workflow with schema and tests - Introduce daily paper schema types (DailyBriefOutput, PaperInfo, PaperNoteOutput, etc.) - Create daily paper cookbook module with analyze, collect, digest, rank, and select steps - Add cookbook entry point and integrate into main steps module - Replace job config export with daily brief output in schema exports - Add comprehensive unit tests covering pipeline, filtering, and output generation - Update dependencies including openai-codex and pypdf packages - Configure standalone daily paper cron job with proper scheduling and routing * test(daily_paper): update tests to use Claude Code wrapper exclusively - Add test to verify web search is disallowed by default in Claude Code - Update imports to include DailyBriefOutput, PaperNoteOutput, and PaperSelection schemas - Change test name from standalone_config_has_backend_split to reflect Claude Code only usage - Remove default agent wrapper and configure all steps to use Claude Code wrapper - Rename select_wrapper to cc_wrapper for clarity and consistency - Remove duplicate Claude Code wrapper initialization - Update test assertions to verify output schema usage matches expected sequence - Remove unused as_llm component from standalone configuration test * refactor(agent-wrapper): simplify skill resolution logic across all wrappers - Replace duplicate skill resolution code with centralized _resolve_project_skills method - Add project_path property with configurable relative path resolution - Introduce proper validation for skill names and directory existence - Change Codex wrapper to use project_path instead of workspace_path for skills - Add SKILL.md requirement validation for project skills - Remove redundant skill processing logic from individual wrappers * feat(daily_paper): add daily paper workflow with PDF analysis and brief generation - Implement shared state management and file helpers for daily-paper steps - Add PDF download and text extraction capabilities with arXiv integration - Create paper collection step with Hugging Face weekly/monthly rankings - Build ranking system using reciprocal-rank fusion with memory keyword scoring - Add Claude Code integration for paper analysis and detailed note generation - Implement digest step to create final five-minute brief from detailed notes - Add configuration for standalone daily cookbook application with cron scheduling - Create typed schema for paper information, selection, and output formats - Add atomic file writing with temporary file safety mechanisms - Implement exclusion logic for previously recommended papers and daily filters * feat(daily_paper): add DingTalk notification integration and enhance logging - Integrate DingTalk markdown send step to notify groups about daily paper briefs - Add comprehensive logging throughout daily paper workflow including start/finish events - Update daily paper analysis prompt to include code repository context requirement - Configure DingTalk notification in daily_cookbook.yaml with app credentials - Add dingtalk-stream dependency for proactive message API integration - Enhance daily paper README with DingTalk notification section and updated flow chart - Implement detailed logging for each step including paper processing and agent calls - Add test coverage for DingTalk markdown sending functionality and configuration - Update pre-commit config to exclude skills directory from checks - Add .claude/skills to gitignore for local development environment * refactor(dingtalk): move dingtalk_stream import to local scope and improve code safety - Moved global dingtalk_stream import to local scope in send.py to avoid eager loading - Added dynamic import with error handling for optional dependency cases - Updated test suite to verify lazy loading behavior works correctly - Fixed markdown title generation by using safe variable naming in wait.py - Enhanced test coverage for arxiv PDF download caching functionality - Updated application context initialization with proper resource directory configuration - Modified paper metadata to include source PDF path reference in output files * refactor(daily_paper): remove manifest system and store selection metadata in digest files - Remove JSON manifest creation and storage functionality - Store selection data directly in digest file frontmatter instead of separate manifest files - Add load_saved_selection method to rebuild selection from digest and paper-note metadata - Update README documentation to reflect new cookbook workflow architecture - Modify test cases to verify selection metadata in digest files instead of manifest JSON - Remove unused json import from multiple daily paper modules - Integrate PaperSelection schema for proper data validation in stored metadata * docs(daily_paper): add bilingual cookbook guides	2026-07-22 19:17:01 +08:00
xyf2020	630f26b119	feat(search): scoped dedup, session-chunk merge, and unified recall formatting (#384) ... * feat(search): add tool_context-scoped chunk dedup with TTL Introduce _ToolContextDedupMixin shared by search/vector_search/bm25_search to skip already-seen chunks within one agent tool_context. Per-context state lives in app_context.metadata with configurable TTL (default 24h). * feat(search): unify chunk answer rendering with merge and explicit empty messages - Refactor SearchStep/VectorSearchStep/Bm25SearchStep to share format_chunks_answer for consistent source rendering and adjacent session-chunk merging. - Distinguish empty results: ALL_RETURNED_MESSAGE when dedup removes everything vs NO_RESULTS_MESSAGE when nothing matched. - Bump JsonlFileChunker default max_chars to 4000. - Add unit tests for source-format merge and empty-result messages. * refactor(config): reorganize file_chunker components and move jsonl max_chars into config - Register explicit markdown/json/jsonl chunkers in beam.yaml and lme.yaml with markdown options (embed_toc, max_ast_sections, frontmatter handling) and jsonl max_chars=4000. - Restrict default chunker to txt/log extensions. - Revert JsonlFileChunker code default max_chars back to 2000; the 4000 value now lives in config. * chore(benchmark): increase longmemeval num_items from 64 to 500 * refactor(search): split SearchStep into simplified and v2 variants, extract counter utility - Extract global_counter_next from ApplicationContext into reme/utils/counter.py as a standalone function operating on metadata dict with lazy initialization. - Split SearchStep into two variants: - SearchStep (simplified): inline chunk.id dedup, single-branch vector/keyword optimization based on vector_weight, inline answer formatting. - SearchV2Step (full): preserves _ToolContextDedupMixin with interval-subset-aware dedup and format_chunks_answer with session-aware chunk merging. - Update beam.yaml and lme.yaml to use search_v2_step for benchmark jobs. - Rename existing search tests to test_search_v2_step_* and add new test_search_step_* tests covering the simplified variant. * fix: normalise missing trailing newline in _build_union_chunk to prevent line collision * refactor: lazy-init counter tree in ApplicationContext metadata - Remove hardcoded _counter_tree and _counter_tree_lock initialization from ApplicationContext.metadata; rely on lazy initialization in reme.utils.counter.global_counter_next on first call - Set longmemeval num_items back to 500 - Remove obsolete trailing-newline collision tests --------- Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>	2026-07-22 17:17:23 +08:00
xyf2020	7b1da5a9ee	feat(benchmark): add BEAM & restructure LongMemEval evaluation framework (#375) ... * feat(eval): add LongMemEval evaluation framework with tool_defaults date injection - Add evaluation/longmemeval/ with run.py, config.yaml, and test scripts - Add reme/config/longmemeval.yaml for evaluation-specific model config - Add tool_defaults mechanism to as_agent_wrapper for injecting default tool kwargs (uses setdefault so LLM-provided values take priority) - Pass tool_defaults={'daily_write': {'date': day}} in auto_memory to ensure notes always use the correct historical date - Add timestamp interpolation (_interpolate_timestamps) in auto_memory for filling missing created_at fields via linear interpolation - Evaluation pipeline: ingest sessions -> dream -> search -> answer -> judge - Uses qwen3.6-flash for memory, qwen3.7-max for answer/judge * chore: gitignore logs/results/demo.py, keep empty dirs * chore: update .gitignore * feat(eval): add multiprocessing and session time filtering to longmemeval runner - Replace async execution with synchronous + multiprocessing for parallel item evaluation - Add filter_future_sessions option to only ingest sessions <= question date - Add question_types filtering in config - Add result summary with binary accuracy and avg score - Update config defaults (oracle variant, 50 items, 32 workers) - Minor code style fixes in agent_wrapper and auto_memory * feat: add bench_query_step with ReAct agent for benchmark query phase - Add BenchQueryStep using agent_wrapper with search job tool - Replace manual search+LLM answer in run.py with bench_query_job - Remove unused answer LLM config from longmemeval.yaml - Register benchmark step module in steps/__init__.py * feat: add start_date/end_date time filter support for search job - Add _extract_date_from_path to extract validated YYYY-MM-DD from chunk paths - Add start_date/end_date filtering in _matches_search_filter - Implement progressive recall in FaissLocalFileStore.vector_search - Promote start_date/end_date from context to search_filter in SearchStep - Add start_date/end_date parameters to search job in default.yaml - Add unit tests for date filter functionality * fix: validate/normalize date filters and harden _extract_date_from_path Address three code-review comments on the time_filter search feature: 1. Validate/normalize start_date and end_date before string comparison. _matches_search_filter does lexicographic comparison against path_date (always canonical YYYY-MM-DD). Raw caller values like '2026-2-28' or 'abc' would produce silently wrong results. Now SearchStep normalizes valid dates via extract_daily_date (with strptime fallback for non-zero-padded input) and silently ignores invalid dates with a logger.warning, removing them from the filter. 2. Clarify behavior for paths without embedded dates. Added optional strict_date_filter parameter (default False). When True and at least one date bound is active, chunks whose path yields no date (e.g. digest/personal/topic.md) are excluded. When False (default), the existing behavior is preserved — dateless paths pass through. 3. Harden _extract_date_from_path against non-standard suffixes. Previously parts[1].split('.')[0] accepted '2026-05-18.anything' as a valid date. Now only exact 'YYYY-MM-DD' (dir) and 'YYYY-MM-DD.md' (day-index) forms are accepted. * feat(eval): LLM-as-Judge per-type prompt routing, binary-only, progress tracking - Remove 0-5 score metric, keep only binary (yes/no) classification - Load per-question-type judge prompts from llm-as-judge.json (temporal-reasoning, knowledge-update, single-session-preference, __default__) - Replace SCORE_JUDGE_PROMPT with type-specific BINARY_JUDGE_PROMPT template - judge_response(): parameter 'metric' -> 'question_type', returns single 'judgment' - Summary output: add per-type accuracy breakdown, remove score stats - Add progress tracking: background thread prints PROGRESS every 10min - Add FINAL progress line and total elapsed time on completion - Add --log-level, --reme-log-level, -q CLI arguments - Parallel mode: pool.map -> pool.imap_unordered for real-time progress - config.yaml: full oracle (10000 items), 32 workers, all question types - Add kill.sh (process cleanup) and run_async.sh (background eval launcher) * docs: add LongMemEval oracle evaluation results (61.6% accuracy) * feat(bench): add MAX_ITERATION limit to BenchQueryStep and add _auto_memory.yaml * feat: add golden session benchmark & eval_only mode with refined prompt - Add benchmark/longmemeval/run_golden_session.py for golden session evaluation - Refine PROMPTED_SYSTEM_PROMPT: concise answer rule, remove 'Information not found' fallback - Add eval_only mode to run.py (--eval_only flag) - Add multiple eval config variants (evalonly, full, test5) - Add analyze_results.py for result parsing - Update auto_memory.yaml, longmemeval.yaml, application_config - Update result-longmemeval.md with latest evaluation results - Add benchmark results to .gitignore * update: refine answer prompts and increase max iteration to 6 - Tighten prompted-answer system prompt for more concise output - Comment out 'Information not found' fallback rule - Increase MAX_ITERATION from 5 to 6 in bench_query - Add recall_eval.py - Update evaluation results * feat(chunker): add dedicated JSON and JSONL file chunkers (cherry-pick from upstream #325) - Add JsonFileChunker: structure-aware chunking preserving nested key paths, optional list-to-dict conversion, size measured by json.dumps() char count - Add JsonlFileChunker: line-aligned sliding-window chunking with configurable overlap, supports char/byte mode switching - Register both chunkers in default.yaml (json for .json, jsonl for .jsonl) - Add comprehensive unit tests (21 + 20 test cases) * feat(service): add CLI service for local job execution (from upstream #334) - Introduce CliService to execute single jobs locally without serving ports - Add prepare_start_config and should_precheck_start functions for CLI job setup - Update reme start command to use CLI service when job argument is provided - Add show_metadata to client kwargs for optional CLI metadata output - Add unit tests for CLI service functionality and configuration handling * feat(steps): add BM25/vector search steps, Python execute step, and draft steps (from upstream #334) - Add Bm25SearchStep for plain BM25 keyword search with tool_context deduplication - Add VectorSearchStep for plain vector search with tool_context deduplication - Add PythonExecuteStep to run Python code in subprocess with timeout handling - Add AddDraftStep/ReadAllDraftStep for draft accumulation scoped by tool context - Update SearchStep with tool_context dedup, dynamic default limit via REME_SEARCH_LIMIT env, and candidate_multiplier default changed from 3.0 to 5.0 - Add comprehensive unit tests for all new steps * feat(search): add tool context deduplication and improve search configuration (#321) * feat(search): add tool context deduplication and improve search configuration - Modify _make_tool methods to accept and inject tool_context_id parameter - Add tool_context_id handling in AS and CC agent wrappers - Increase search candidate multiplier from 3.0 to 5.0 in default config - Extend HTTP client timeout from 30s to 3600s - Add tool context deduplication logic to prevent duplicate search results - Implement TTL-based expiration for seen chunks in tool contexts - Add comprehensive unit tests for tool context deduplication behavior - Update .gitignore to exclude longmemeval directory - Add time import for timestamp functionality in search step * refactor(search): replace time module with datetime for timestamp generation - Removed unused time import - Added static method _now_ts using datetime.timestamp - Updated clock parameter to use _now_ts method instead of time.time - Maintained same timestamp precision and functionality * fix(file_io): fix risk of out-workspace paths (#322) * fix(file_io): fix risk of out-workspace paths * chore(file_io): remove unused unittest file * fix(as_embedding): support both agentscope 2.0.2 and 2.0.3 (#323) 2.0.3 promoted `dimensions` to a required first-class constructor argument while keeping a backfill from `parameters.dimensions`; 2.0.2 has no such argument and reads `dimensions` from `Parameters`. Keep `dimensions` in `Parameters` for both versions and, when the model constructor accepts `dimensions`, pass `dimensions=None` so 2.0.3's backfill promotes it out of `parameters`. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * Bump version to 0.4.0.7 * refactor: delegate LLM-as-Judge to answer_judge_step and update eval config/results - run.py: replace inline judge logic with judge_response_via_job using app.run_job('answer_judge') - longmemeval.yaml: expand benchmark configuration - bench_query.py: enhance benchmark query step - result-longmemeval.md: update evaluation results - judge_all_plus_results.json: add judge all-plus results * refactor: split longmemeval.yaml into lme.yaml/beam.yaml and unify job names - Split reme/config/longmemeval.yaml into lme.yaml (LongMemEval) and beam.yaml (BEAM) - Unify job names across both configs: agentic_answer, answer_judge, context_answer - Update evaluation/longmemeval/run.py and evaluation/beam/run_beam_eval.py to use unified job names - Update all evaluation config YAMLs to reference lme.yaml - Add BEAM benchmark step implementations (agentic_answer, context_answer, llm_judge) - Remove obsolete config_test5.yaml and test_5sessions.py * eval: BEAM 100K & LongMemEval cleaned-S 评测结果记录 - BEAM 100K eval-only (32并发, 20 case): Agentic 0.631, Prompted 0.468 - LongMemEval final GT (500题): Agentic 89.0%, Prompted 83.6% - 新增 benchmark/result-beam.md, benchmark/result-longmemeval.md - benchmark/beam/config.yaml: num_workers=32 * refactor: restructure benchmark directory and clean up gitignore rules - Consolidate benchmark outputs to benchmark/results/ with .gitkeep - Remove old benchmark scripts, configs and result files from benchmark/beam/ and benchmark/longmemeval/ - Add datasets/README.md and datasets/README_EN.md with download instructions - Add datasets/longmemeval/download.py and final_groundtruth_cleaned_s.json - Add memory_workspaces .gitkeep placeholders - Restructure .gitignore: fix duplicate entries, add BEAM dataset exclusion, refine logs/results ignore patterns - Remove stale result-beam.md and result-longmemeval.md from project root * chore: clean up longmemeval benchmark scripts and update dataset docs - Remove obsolete longmemeval benchmark runner/stats scripts - Update datasets/longmemeval README and add Chinese translation - Clean up final_groundtruth_cleaned_s.json * docs(benchmark): add reproduction guide for LongMemEval and BEAM - Add bilingual README for benchmark runners (EN/ZH) - Cover prerequisites, dataset download, run commands, configs, outputs, logs, and kill.sh * refactor: migrate auto_memory steps from evolve to benchmark-specific modules - Split auto_memory into beam and lme benchmark-specific implementations - Add auto_memory.py and auto_memory.yaml under steps/benchmark/beam and steps/benchmark/lme - Slim down evolve/auto_memory.py and auto_memory.yaml to shared base only - Remove obsolete evolve/_auto_memory.yaml - Update benchmark run.py, config YAMLs, and step __init__.py registrations - Update llm_judge and context_answer minor adjustments - Remove outdated test_lme_final_answer_review.py * revert(as_agent_wrapper): sync with upstream/main Remove local-only comment to keep file identical with upstream/main. * style: add trailing commas in benchmark __init__.py __all__ lists * chore: disable vector_weight range assertion in SearchStep * chore: add tests/integration/logs/ to .gitignore * refactor: replace scipy.stats.kendalltau with pure numpy implementation scipy is not listed in project dependencies. Implement Kendall's tau-b rank correlation using only numpy to remove the undeclared dependency. * feat(benchmark): add binary score metrics, update BEAM 1M results, and improve LLM retry/prompt config - benchmark/beam/run.py: add binary score calculation per rubric item and per-type/overall binary stats - benchmark/beam/config.yaml: switch to 1M dataset, reduce workers to 18 - benchmark/result-beam.md: add 1M evaluation results with binary scores - benchmark/result-longmemeval.md: minor formatting - reme/config/beam.yaml: increase max_retries to 5 and add retry_delay 5.0 for all LLM components - reme/config/lme.yaml: increase max_retries to 5 and add retry_delay for judge/prompted/bench components - reme/steps/benchmark/lme/agentic_answer.yaml: improve search strategy and answer rules prompts * fix(benchmark): fix line-too-long and add pylint disable for main() * refactor(longmemeval): use single cleaned-S dataset with embedded ground truth - Switch to agentscope-ai/ReMe_longmemeval_clean_s_v2 HuggingFace source - Remove separate final_groundtruth_cleaned_s.json (ground truth now in data file) - Simplify download.py to fetch only longmemeval_s_reme_cleaned.json - Remove dataset.variant and dataset.ground_truth_path config options - Update benchmark and datasets READMEs to reflect new workflow - Update .gitignore for new dataset filename * fix: rename loop variable to avoid pylint redefined-outer-name warning * refactor(benchmark): restructure datasets/memory_workspaces into benchmark and simplify auto_memory steps * refactor(benchmark): extract BaseAgenticAnswerStep into base module - Add reme/steps/benchmark/base/agentic_answer.py with shared agentic answer logic - Refactor beam/lme AgenticAnswerStep to inherit from BaseAgenticAnswerStep - Simplify lme/context_answer.py and update context_answer.yaml - Update result-longmemeval.md with latest evaluation results (agentic 91.0%) * refactor(benchmark): remove context_answer steps and unused configs - Remove beam/lme context_answer job definitions and step implementations - Remove prompted LLM component from beam.yaml and lme.yaml - Delete jinli_lme.yaml (no longer needed) - Simplify benchmark run.py scripts - Clean up .gitkeep files and update .gitignore - Remove unused import in search.py * chore: remove benchmark/results/.gitkeep --------- Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135> Co-authored-by: jinliyl <6469360+jinliyl@users.noreply.github.com> Co-authored-by: imrewce <wce@pku.edu.cn> Co-authored-by: Sen Huang <48879559+ployts@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-21 19:09:50 +08:00
jinliyl	e7d44f6f3b	refactor(agent): unify agent subprocess env, sessions, skills, and MCP/service jobs (#382) ... * feat(config): add environment variable configuration for agent subprocesses - Add environment field to ApplicationConfig to store variables for agent subprocesses - Remove dynamic loading of .env files in agent wrappers - Introduce subprocess_environment property in base agent wrapper - Pass application-level environment variables to Claude Code and Codex agents - Load environment variables once at startup and pass to ReMe application - Remove dependency on load_env utility in agent wrapper implementations - Update tests to use configured environment instead of dynamic loading - Remove unused environment loading utilities and related test cases * refactor(mcp): remove channel notification system and related components - Removed channel notification step implementation - Removed claim channel step implementation - Removed ChannelSink class from MCP service - Removed channel-related documentation from AGENTS.md - Removed channel instruction text from MCP service - Removed all channel-related tests - Updated application context metadata comment to remove channel sink reference - Removed channel module initialization and imports * feat(service): add job whitelisting capability to BaseService - Add optional jobs parameter to BaseService.__init__ to configure job whitelist - Store jobs as set in self.jobs attribute for efficient lookup operations - Modify add_jobs method to filter jobs based on whitelist configuration - Update documentation in both English and Chinese to describe new feature - Add comprehensive unit tests for job whitelisting behavior - Implement flowchart update showing new filtering logic - Preserve existing enable_serve flag behavior alongside new whitelisting * refactor(service): enhance service job validation and MCP tool injection - Add strict validation for service jobs whitelist with detailed error messages - Implement injected job arguments support for MCP services with conflict detection - Add tool error handling for unsuccessful responses in MCP services - Remove duplicate job names in Codex agent wrapper using dict.fromkeys - Update MCP server argument format from single JSON array to repeated --job flags - Add comprehensive test coverage for job injection and error handling scenarios - Update documentation to reflect service job validation and MCP features - Ensure application cleanup occurs even when service lifespan encounters errors * feat(agent): update skill handling to preserve existing Claude skills - Change skills parameter processing to use 'all' instead of filtered list - Add logic to select project skills without restricting Claude's existing skills - Update variable naming from 'skills' to 'selected_skills' for clarity - Modify application context metadata documentation to clarify in-memory state usage - Add test case to verify configured skills are added without filtering existing skills - Update internal skill directory handling to use renamed variable consistently * refactor(agent): restructure agent wrapper components and session storage - Move CcFileSessionStore to separate module for better organization - Add SDK package version logging in base agent wrapper - Update Claude Code agent to use new session store structure with project keys - Refactor Claude Code agent wrapper to use proper type hints and SDK integration - Add support for server tool use events in Claude Code message processing - Improve error handling and resource cleanup in streaming operations - Update Codex agent wrapper with proper type annotations and configuration - Remove deprecated system prompt mode handling from Claude Code wrapper - Fix session path construction for Claude Code transcript storage - Update dependency injection and configuration handling patterns * fix(cc_agent_wrapper): resolve Claude Code SDK integration issues - Added dataclass import and created _BlockState for content block metadata tracking - Implemented proper MCP server name constant and tool context ID validation - Fixed tool_context_id injection to prevent duplicate assignment errors - Resolved skills parameter handling in build_options method - Enhanced job tools integration with MCP servers mapping validation - Replaced deprecated block_ids/block_types/tool_call_names with block_states dict - Updated message_delta to emit USAGE chunks instead of REPLY_END - Fixed stream result handling to ensure proper REPLY_END emission - Improved error handling for session mirror failures and rate limits - Added proper cleanup for expected trailing errors in streams - Refactored Codex agent wrapper initialization and configuration management - Removed obsolete system_prompt_mode from default config - Enhanced test coverage for new block state and error handling features - Fixed async generator handling with aclosing context manager - Improved chunk type mapping for Claude Code SDK events * refactor(tests): remove demo config tests from config parser test suite - Removed test_demo_config_registers_llm_jobs function and its assertions - Eliminated verification of LLM demo job configurations - Removed checks for agent wrapper component settings - Deleted assertions for model configurations and parameters - Cleaned up deprecated test cases related to demo config parsing * refactor(evolve): simplify Claude Code session store path structure - Removed redundant project key subdirectory from session link generation - Updated CcFileSessionStore initialization to use direct session directory path - Maintained existing session layout compatibility for backward compatibility - Added unit tests to verify session persistence behavior with existing transcripts - Ensured UUID-based session files remain accessible at expected locations - Preserved existing session directory structure without additional nesting * refactor(agent): defer optional Codex SDK imports until first use - Moved openai-codex imports inside functions to avoid mandatory dependencies - Added TYPE_CHECKING guard for development time type checking only - Implemented lazy loading mechanism with _get_async_codex_class function - Updated AsyncCodex initialization to occur on demand rather than at module level - Maintained backward compatibility while improving import performance - Added test case to verify package import works without optional Codex SDK - Updated agentscope dependency to version 2.0.4.post1 in pyproject.toml * test(embedded): add compatibility tests for in-process ReMe embedding - Add test suite for QwenPaw-style embedded configurations - Verify optional defaults remain preserved in embedded configs - Ensure in-process application API stays compatible - Test model injection and lifecycle management compatibility - Remove obsolete hermes agent plugin tests - Update CLI import test to cover multiple optional SDKs - Block claude_agent_sdk and openai_codex during import testing	2026-07-20 23:52:14 +08:00
jinliyl	b4333fbef8	feat(index): add bounded memory-aware batch processing (#381) ... * test(background_steps): add comprehensive tests for batch processing and memory management - Add test for catalog upserts in batches of at most 100 files - Add test for catalog deletes in batches of at most 100 paths - Add test for index memory budget reducing batches to one file - Add test for memory target limiting cumulative batch size - Add test for invalid batch memory settings rejection - Add test for continuing after one batch fails - Add test for yielding to event loop while building batch - Add test for modified file reusing unchanged embedding - Add test for reporting memory estimation failure without aborting feat(update_changes): implement bounded batch processing with memory management - Add configurable batch parameters with default values - Implement memory budget calculation based on available system memory - Add file inspection and memory estimation before processing - Implement batch flushing when limits are reached - Add proper error handling for batch operations - Support async yielding during batch building - Add comprehensive validation for batch configuration parameters - Implement memory estimation for indexing operations - Add batch size limiting for delete operations * test(steps): add tests for memory estimation failure handling - Add test case for isolated file processing when memory estimation fails - Add test case for proper release of flushed items before building next file - Implement weak reference tracking to verify payload lifetime management - Create parametrized tests for both source and item memory estimation methods - Add assertions to verify single-item batch behavior on estimation failures - Include comprehensive error handling verification for memory budget calculations * chore(version): bump version to 0.4.1.3 - Update __version__ from 0.4.1.2 to 0.4.1.3 in __init__.py * feat(index): support batch settings from environment * refactor(index): use direct batch defaults * refactor(index): configure memory estimates through step args * ci: simplify Windows smoke dependencies	2026-07-20 17:25:00 +08:00
Sen Huang	55ef4bd6ad	fix(proactive): expose topics in primary answer (#380)	2026-07-20 16:05:47 +08:00
jinliyl	cf22ef3b1d	feat: add codex auth modes, background embedding/index repair, and qwenpaw logging (#371) ... * feat(codex): add authentication mode support with thread-safe logging - Implement _CodexAuthConfig dataclass for resolved auth settings - Add auth_mode parameter with auto/api_key/oauth options - Separate API key and OAuth authentication flows - Force specific login method based on auth mode - Add explicit API key validation requirement - Serialize concurrent logger initialization in thread lock - Close logging handlers properly during cleanup - Update default config with auth_mode presets for codex and codex_oauth - Add comprehensive tests for authentication modes and concurrent logging * feat(file_store): implement background embedding backfill and keyword index repair - Add _after_embedding_backfill hook in FAISS local file store - Schedule startup embedding repair without delaying component readiness - Cancel and collect embedding backfill task during component shutdown - Log progress at fixed percentage boundaries for long-running operations - Process embedding backfill in configurable batch sizes with progress reporting - Rebuild keyword index in bounded batches with detailed mismatch diagnostics - Format stdlib logs consistently with QwenPaw console output using relative paths - Run embedding backfill as background task that doesn't block component startup - Add comprehensive tests for background embedding and keyword index repair scenarios * fix(file-store): repair graph-chunk consistency on load - Add _repair_graph_chunk_consistency method to detect and fix mismatched graph/chunk states - Clear torn graph/chunk state when missing or orphaned chunks are detected - Ensure keyword index sync handles empty chunks properly - Add comprehensive tests for graph-chunk consistency scenarios - Update test utilities to properly seed graph/chunk snapshots - Increment version to 0.4.1.2 * feat(file_io): enhance list step response format and add comprehensive logging - Format list output with bullet points for better readability - Add explicit "No files found" message when directory is empty - Include detailed timing information for file store startup phases - Add logging for chunk loading, graph consistency checks, and keyword indexing - Provide detailed metrics for embedding backfill operations - Add comprehensive test coverage for empty directory scenarios - Include batch processing statistics for embedding operations * feat(logger): add QwenPaw logging integration with forwarding mechanism - Introduce _ForwardToLoggerHandler to forward log records to target logger - Add qwenpaw logger integration that forwards ReMe logs to QwenPaw handlers - Maintain ReMe logger stability for modules that cache it at import time - Enable QwenPaw handlers to take effect without ReMe reconfiguration - Add comprehensive tests for stdlib forwarding to QwenPaw sinks - Support explicit REME_DISABLE_LOGURU=false to keep original Loguru backend - Preserve existing logging behavior when QwenPaw is not configured * fix(file_store): serialize concurrent FAISS dump operations to prevent corruption - Add asyncio lock to ensure only one FAISS dump operation runs at a time - Generate unique temporary filenames using UUID tokens for atomic replacement - Implement proper cleanup of temporary files in finally block - Add comprehensive test to verify concurrent dumps are serialized - Ensure atomic writes by replacing both index and idmap files together - Prevent partial state writes during concurrent access scenarios	2026-07-20 14:47:34 +08:00
Sen Huang	1c08eaa559	fix: enforce markdown chunk byte limits (#370)	2026-07-17 22:03:58 +08:00
Sen Huang	987f275985	fix: bound markdown chunking for large section trees (#369)	2026-07-17 17:57:56 +08:00
Xinmin Zeng	9c9b040d42	feat(plugins): add Hermes Agent memory provider (#365) ... * feat(plugins): add Hermes Agent memory provider * fix(plugins): harden Hermes memory lifecycle * fix(plugins): keep Hermes writer recoverable	2026-07-17 14:06:12 +08:00
jinliyl	c1a25e9ff4	feat(agent): add Codex agent wrapper and ReMe MCP bridge (#358) ... * feat(agent): add Codex wrapper integration * feat(agent): enhance agent wrapper functionality and add comprehensive testing - Implement structured output schema normalization across all wrappers - Add Claude Code system prompt mode support with append/replace options - Introduce Codex agent wrapper with streaming, tool context isolation, and skill management - Enhance skill linking with validation and conflict resolution - Add approval event streaming support for Codex wrapper - Implement output schema validation and normalize function - Create dedicated test suites for Claude Code and Codex integration - Update README documentation for Codex wrapper capabilities - Refactor kwargs merging with proper schema handling - Add tool context validation when resuming sessions - Implement proper cleanup and session management for Codex wrapper * test(cc-agent): add test coverage for structured output scenarios - Add docstring for empty schema validation in build_options - Document falsy structured output preservation behavior - Add docstring for streaming wrapper schema rejection - Include lambda function reference for wrapper factory consistency - Add test documentation for live Codex wrapper contract exercise * docs: revert README changes * fix(agent): interrupt abandoned Codex turns	2026-07-17 13:39:18 +08:00
jinliyl	329fd9a6a6	refactor(config): remove max_file_bytes limit from background jobs (#367) ... - Removed max_file_bytes configuration from index_update_loop, resource_watch_loop, digest_watch_loop, and reindex jobs - Updated default.yaml to reflect simplified job configurations without file size limits - Removed corresponding test case that validated the 20 MiB limit behavior - Simplified watch directories and suffixes to basic configurations - Cleaned up unnecessary commented configurations in the YAML file	2026-07-17 11:26:39 +08:00
jinliyl	2eb05392c6	chore(benchmark): remove longmemeval final answer review file (#366) ... * feat(benchmark): add final answer review step for evaluation - Introduce FinalAnswerReviewStep to handle answer validation - Add final_answer_review.jsonl dataset with 24 evaluation cases - Include detailed reasoning and golden check results for each case - Support various question types including temporal reasoning and preferences - Implement time consistency checks for session references - Add comprehensive test coverage for different evaluation scenarios * chore(benchmark): remove longmemeval final answer review file - Removed final_answer_review.jsonl containing 23 evaluation records - Deleted question_id mappings with detailed reasoning for golden answers - Removed answer correctness assessments and session time validation checks - Cleaned up benchmark dataset used for memory evaluation testing - Eliminated JSONL format evaluation results for temporal reasoning tasks - Removed references to various session IDs and time-based validations * config(default): disable shell step configuration by commenting out - Commented out the shell step configuration in default.yaml - Disabled asynchronous shell command execution capability - Removed shell step from available backend operations - Preserved traverse backend configuration unchanged * refactor(tests): remove unused shell job test from config parser tests - Removed test_default_config_registers_shell_job function that was no longer needed - Kept existing test for frontmatter chunk metadata configuration - Cleaned up test suite by removing obsolete test case	2026-07-16 20:32:28 +08:00
jinliyl	c3b1e93918	feat(index): add file size limits and oversized file handling (#362) ... * feat(index): add file size limits and oversized file handling - Implement max_file_bytes configuration option for content processing jobs - Add default 20MB file size limit for background processing in default config - Skip oversized files during auto_resource step with appropriate metadata - Clear stale index entries when oversized files are modified - Add size-based filtering logic to update_changes step with skip reporting - Include file size validation in UpdateIndexStep with proper response handling - Add comprehensive tests for oversized file scenarios in auto_resource and update_index - Document file size limits in constants with appropriate thresholds * chore(version): bump version to 0.4.1.1 - Update __version__ from 0.4.1.0 to 0.4.1.1 in __init__.py * fix(index): isolate batch metadata and handle file races	2026-07-15 21:01:18 +08:00
jinliyl	2a85c36fa9	refactor(embedding): defer provider construction until first remote call (#361) ... - Changed dimensions property to avoid forcing provider construction - Added _ensure_model method to construct provider on demand - Modified __call__ to ensure model exists before use - Updated _start to defer provider initialization - Removed eager health check during startup - Added compact embedding serialization with base64 encoding - Implemented batch processing for vector search with heap-based ranking - Added document_ids property to keyword index interface - Updated chunk persistence to handle legacy JSON embeddings - Optimized memory usage by avoiding materialization of metadata in document_ids	2026-07-15 20:46:36 +08:00
jinliyl	2e87b7a52e	feat(core): add shell execution and runtime memory status (#344) ... * feat(core): add shell command execution and memory status reporting - Introduce ShellStep for executing shell commands with timeout support - Add StatusStep to report memory estimates for stateful data components - Register shell and status commands in default configuration - Update documentation with new reme status and shell command capabilities - Implement comprehensive unit tests for both new step types - Add support for asynchronous command execution with proper error handling * feat(config): add log_config option to suppress config loading logs - Add log_config parameter to resolve_app_config function with default True - Conditionally log config loading messages based on log_config flag - Update reme.py and service_utils.py to use log_config=False for client calls - Suppress config logging in user-facing contexts to avoid output pollution refactor(shell): rename command parameter to cmd for clarity - Change 'command' to 'cmd' in default.yaml configuration schema - Rename 'timeout' to 'shell_timeout' to avoid parameter name collisions - Update ShellStep to accept both legacy and new parameter names - Maintain backward compatibility with existing command/timeout usage test(shell): add comprehensive tests for shell step parameter handling - Add test cases for new cmd and shell_timeout parameter names - Verify legacy command and timeout parameters still work - Test blank command rejection message updated to use cmd - Create integration test for shell parameter payload passing * fix(shell): ensure proper environment loading and process timeout handling - Move load_env() call to execute before parse_args() in main function - Add proper process group killing for timeout scenarios on POSIX systems - Implement recursive child process termination on Windows for proper cleanup - Change parameter name from 'timeout' to 'shell_timeout' in shell execution - Remove support for legacy 'command' and 'timeout' parameter names - Update test cases to verify new timeout behavior and parameter requirements - Add comments explaining component size tracking implementation details	2026-07-14 16:31:41 +08:00
jinliyl	bf7ca17705	feat(benchmark): add LongMemEval golden answer validation (#335) ... * feat(benchmark): add golden answer validation and session review for LongMemEval - Introduce GoldenCheckStep to validate LongMemEval golden answers using structured verdicts - Add SessionReviewStep to extract query/answer-relevant evidence from all sessions - Implement concurrent session processing with configurable concurrency limits - Create check_golden job configuration with lme_review and lme_judge agent wrappers - Add Qwen3.7-plus model configuration for enhanced processing capabilities - Include python_execute tool integration for agent-based reasoning and date validation - Generate comprehensive JSON output with session summaries and validation verdicts - Add run_check_golden.py script for batch processing across all LongMemEval samples - Configure proper logging initialization with console and file output options - Update component registry and file I/O modules to support new benchmark features * feat(scripts): add script to summarize LongMemEval check_golden verdicts - Parse check_golden.json files across all LongMemEval samples - Calculate accuracy metrics for golden answers and session IDs - Provide breakdown by question type with percentage calculations - Add command line options for listing bad samples and JSON output - Include progress tracking showing completed vs pending samples - Display confidence scores and date sanity checks statistics * refactor(benchmark): move golden check scripts to longmemeval directory - Moved run_check_golden.py from scripts/ to benchmark/longmemeval/ - Moved stats_check_golden.py from scripts/ to benchmark/longmemeval/ - Updated path resolution to use parents[2] instead of parent.parent - Added new --list-run-failed option to stats script - Added logging directory constant and functions for tracking launched samples - Enhanced stats output with launched count and run failure information - Improved error reporting with run failure details and log file paths * feat(benchmark): add LongMemEval agentic answer workflow with session extraction - Add LmeAgenticAnswerStep, LmeAutoMemoryStep, and LmeExtractSessionStep to __init__.py - Create shared helper render_with_source for displaying search results with session_id - Implement agentic_answer step with vector_search, bm25_search, and extract_session_by_id tools - Add auto_memory step to convert each session into search-friendly daily notes - Create extract_session step to retrieve and analyze raw session content by session_id - Update jinli_lme.yaml with auto_memory, vector_search, bm25_search, and agentic_answer jobs - Configure lme_memory, lme_extract, and lme_agentic_answer agent wrappers - Enhance search steps with include_source option to show session_id metadata - Add proper session_id tracking and collision handling in daily note generation * feat(benchmark): add LongMemEval agentic answer evaluation pipeline - Add session_id tracking to agentic_answer.py result metadata - Introduce run_agentic_answer.py driver for complete pipeline execution - Implement auto_memory, update_index, and agentic_answer job orchestration - Add concurrent execution with configurable limits and staggering - Create aggregation script for collecting tool-call trails and results - Add stats_agentic_answer.py for comprehensive result analysis - Implement resume capability with existing output detection - Generate aggregate.json with per-sample breakdown and tool call summaries * feat(steps): add ClearPathsStep for cleaning workspace outputs before rebuild - Introduce ClearPathsStep to remove stale workspace files/directories - Add support for specifying paths and config_keys as targets to clear - Implement safety checks to prevent deletion of files outside workspace - Add logging for cleared paths and warnings for invalid paths - Configure clear_paths_step in jinli_lme.yaml to clean daily_dir - Add clear_paths_step to clean mem_answer.json before rebuilds * feat(benchmark): add resume functionality to agentic answer runner - Replace --force flag with --resume flag for controlling job execution - By default every job reruns with clean rebuild behavior using config clear steps - Add --resume option to skip samples whose output already exists and continue interrupted batches - Update documentation to reflect new default clean rebuild behavior - Modify job skipping logic to honor resume flag instead of force flag - Update dry-run output to show correct todo jobs based on resume status - Change default example command to use --resume for continuing interrupted runs * feat(benchmark): generate JSONL output for check golden records - Add write_check_golden_list function to create JSONL file - Write all readable check_golden records as JSONL format - Include check_golden_list path in stats output - Display generated JSONL file path in summary report - Maintain UTF-8 encoding with non-ASCII character support * refactor(benchmark): rename answer judge step and integrate LME LLM judge - Rename AnswerJudgeStep to LmeLlmJudgeStep and update imports - Add new llm_judge configuration in jinli_lme.yaml - Update run_agentic_answer.py to include llm_judge in pipeline - Modify LmeLlmJudgeStep to read from query.json and answer.json - Write LLM judgement results back to mem_answer.json - Add command line options for start/end sample range selection - Update aggregate.json generation to include LLM judgement data - Add resume capability for llm_judge job based on judgement presence * refactor(benchmark): rename answer judge step and integrate LME LLM judge - Rename AnswerJudgeStep to LmeLlmJudgeStep and update imports - Add new llm_judge configuration in jinli_lme.yaml - Update run_agentic_answer.py to include llm_judge in pipeline - Modify LmeLlmJudgeStep to read from query.json and answer.json - Write LLM judgement results back to mem_answer.json - Add command line options for start/end sample range selection - Update aggregate.json generation to include LLM judgement data - Add resume capability for llm_judge job based on judgement presence * feat(steps): add wait_for_paths_step to block until workspace files exist - Introduce WaitForPathsStep class that polls for required workspace-relative paths - Add step registration with 'wait_for_paths_step' backend identifier - Implement path validation to ensure targets are within workspace boundaries - Add polling mechanism with configurable intervals via poll_seconds parameter - Include logging functionality with log_every_seconds parameter for status updates - Add metadata tracking of waited paths and duration in response object - Register step in index module and expose in public API - Configure step in jinli_lme.yaml to wait for session_review.json before golden check - Add script rename from run_check_golden.py to run_golden_check.py with enhanced options * feat(benchmark): enhance longmemeval benchmarking with concurrency and progress tracking - Add benchmark extra dependency group with portalocker requirement - Introduce concurrent execution support for golden_check and session_review workflows - Add progress reporting interval option with real-time status updates - Implement global throttling mechanism for session review requests using file locks - Enhance golden check validation with current schema verification - Add active task tracking and graceful shutdown handling - Rename check_golden scripts to golden_check for consistency - Update statistics reporting with correct/incorrect terminology instead of reasonable - Add stale format detection and compatibility handling for verdict fields - Include both_correct rate calculation in accuracy metrics - Add concurrency and staggering options for better resource management * ci(workflow): add Windows smoke test workflow - Create new workflow file .github/workflows/windows-smoke.yml - Configure workflow to trigger on push and pull request events - Set up Python environment with version 3.11 - Install package dependencies using pip - Run version job as smoke test for CLI functionality - Enable concurrency control to prevent duplicate runs - Use matrix strategy for Python version testing * feat(benchmark): add retry mechanism and health check for session review - Added retry configuration options (retry_initial_seconds, retry_max_seconds, retry_max_attempts) to jinli_lme.yaml - Implemented exponential backoff retry logic with configurable parameters in session_review step - Added output_is_healthy function to verify session_review.json integrity and absence of failed reviews - Updated resume functionality to skip only healthy outputs instead of all existing files - Integrated JSON parsing and validation to check for failed reviews in output files - Enhanced error handling and logging for retry attempts and recovery scenarios * feat(benchmark): add LongMemEval session review statistics script - Create stats_session_review.py to summarize session_review.json artifacts - Add command line options for listing failed, missing, and run failed samples - Implement JSON output mode for programmatic consumption - Calculate and display health statistics including total samples, healthy outputs, failed sessions - Provide detailed failure information with session IDs and error messages - Generate re-run commands for samples with failed reviews - Add percentage calculations for better statistical overview - Include support for multiple output formats and detailed logging * feat(benchmark): add LongMemEval output cleanup script and enhance golden check retry logic - Added clean_sample_outputs.py script to remove generated LongMemEval files while preserving source inputs - Implemented configurable retry mechanism in golden_check.py with exponential backoff strategy - Added retry parameters (initial/max seconds and max attempts) to control failure recovery behavior - Integrated asyncio support for asynchronous sleep during retry intervals - Configured default retry settings in jinli_lme.yaml with 5s initial and 300s maximum intervals - Preserved core files (query.json, answer.json, session/) while cleaning generated artifacts * feat(benchmark): add AppleDouble file cleanup to sample output cleaner - Remove AppleDouble files starting with '._' recursively including under session/ - Add is_under helper function to check if path is inside parent directory - Track targets in set to avoid duplicate processing - Include AppleDouble files in cleanup targets when not already covered by existing targets - Maintain dry-run mode as default behavior with --apply flag for actual deletion * refactor(benchmark): update LongMemEval sample output cleaning script - Add time and Iterator imports for enhanced functionality - Add --progress-every argument to control progress reporting frequency - Replace is_under function with iter_sample_targets generator - Implement detailed progress tracking with timing measurements - Add sample-by-sample processing with elapsed time reporting - Include AppleDouble file detection within session directory - Update target counting and deletion statistics display - Add conditional progress updates based on progress-every setting - Improve dry-run mode with would-delete indication * chore(benchmark): increase initial interval for session review step - Changed START_INTERVAL_SECONDS from 1.0 to 3.0 seconds - Adjusted timing parameters for better benchmark stability * refactor(benchmark): implement coordinated retry mechanism for session reviews - Add retry gate condition to coordinate concurrent review attempts - Implement wait_for_healthy_start_slot to handle sequential retries - Create mark_retrying and mark_recovered functions to track retry states - Update reply_with_retry to accept index parameter for coordination - Add has_prior_retry logic to prevent race conditions during recovery - Ensure proper cleanup of retry state on success or failure - Maintain backward compatibility while adding coordination features * chore(benchmark): adjust session review start interval timeout - Changed START_INTERVAL_SECONDS from 3.0 to 5.0 seconds - Increased initial delay for session review benchmark step - Updated timeout configuration for improved stability * refactor(benchmark): update session review concurrency and throttling mechanism - Replace global throttle with per-process concurrency control - Add concurrency parameter with default value of 30 in config - Add start_interval_seconds parameter with default value of 2 seconds - Change default concurrency from 3 to 1 in command line interface - Update documentation to reflect new throttling behavior - Implement semaphore-based concurrency limiting for review tasks - Modify retry mechanism to use local locking instead of global files - Remove portalocker dependency for cross-process throttling * refactor(config): update session review configuration and concurrency settings - Removed deprecated retry configuration parameters from jinli_lme.yaml - Increased MAX_CONCURRENCY from 30 to 60 in session_review.py - Reduced START_INTERVAL_SECONDS from 2.0 to 1.0 in session_review.py - Cleaned up redundant backend specifications in configuration file - Simplified agent wrapper configurations by removing obsolete retry settings * feat(benchmark): enhance LME auto memory step with advanced scheduling and error handling - Add datetime parsing functionality for LongMemEval timestamps with regex pattern - Implement configurable concurrency limits with MAX_CONCURRENCY of 60 - Introduce retry mechanism with exponential backoff for agent interactions - Add session filtering based on date comparison with question_date validation - Create rate limiting with start interval control between requests - Implement sophisticated retry coordination using asyncio conditions - Add comprehensive error tracking for failed and filtered session extracts - Remove deprecated concurrency parameter from jinli_lme.yaml configuration - Add structured output validation in session review step - Include detailed metadata reporting with session statistics and errors * fix(benchmark): adjust default concurrency for auto_memory job - Changed default concurrency from 3 to 1 for auto_memory job to prevent API overload - Updated help text to reflect new default value of 1 for concurrency parameter - Modified documentation to clarify concurrency behavior varies by job type * refactor(search): replace hardcoded candidate multiplier with constant - Introduced _CANDIDATE_MULTIPLIER constant set to 10 - Replaced hardcoded factor of 5 with _CANDIDATE_MULTIPLIER in BM25 search - Replaced hardcoded factor of 5 with _CANDIDATE_MULTIPLIER in vector search - Updated test to verify both search steps use ten times limit for candidates - Imported VectorSearchStep and Bm25SearchStep in test module - Added comprehensive test case for candidate count calculation logic * feat(lme): add data inspection error handling with fallback mechanism - Implemented non-retryable data inspection error markers detection - Added _is_data_inspection_error method to identify inspection failures - Created fallback handling for data inspection errors in auto memory extraction - Added fallback handling for data inspection errors in session review - Extended failed extracts tracking with non-retryable and fallback flags - Separated fallback extracts from regular failed extracts in reporting - Enhanced error logging with specific data inspection failure messages - Updated metrics to track fallback extractions and reviews separately - Maintained existing retry logic for other exception types * feat(benchmark): enhance session review statistics with fallback tracking - Add support for identifying and listing non-retryable fallback reviews - Introduce --list-fallback argument to display fallback review details - Separate retryable failures from non-retryable fallbacks in reporting - Track fallback samples and sessions separately from failed ones - Update console output to show both retryable and non-retryable categories - Include fallback details in JSON output with reasons and session info - Modify failure counting logic to distinguish between retryable and fallback reviews * feat(benchmark): add question_id tracking and enhanced fallback reporting - Add question_id function to extract query.question_id from data - Initialize question_id_by_id dictionary to store question IDs by index - Store question_id for each sample during data processing - Enhance fallback output to include question IDs and session information - Format sample labels with question IDs when available - Display session IDs associated with each fallback case * feat(benchmark): add question_id support and improve bad sample reporting - Add question_id_for function to extract question_id from multiple sources - Add sample_label function to format samples as idx(question_id) when available - Store question_id in data dictionary during processing - Change bad_golden and bad_sessions to store full records instead of just indices - Update list_bad output to show formatted labels with question_id information - Improve error reporting with more detailed sample identification * feat(benchmark): enhance golden check stats with structured output - Add related_session_ids function to extract session IDs from verdict records - Create grouped_records function to group records by question type - Replace flat list output with JSON-formatted grouped records in list_bad option - Replace flat list output with JSON-formatted grouped records in list_bad_sessions option - Maintain Chinese labels while adding structured data presentation - Improve readability of bad verdict record display with hierarchical grouping * feat(benchmark): update data structure for question indexing - Replace sample_label with _idx field for index tracking - Add question_id field to store _question_id values - Maintain backward compatibility with empty string defaults - Preserve existing session_id functionality - Update data mapping to include new fields in grouped results * refactor(benchmark): streamline golden answer verification process - Replace relevance filtering with comprehensive information extraction - Remove is_relevant field and simplify session summary structure - Change relevant_info to extracted_info for clarity - Update golden check logic to work with full extractions instead of filtered summaries - Simplify prompt instructions to focus on complete information extraction - Remove redundant schema validation and structured output requirements - Adjust statistics calculation to match new extraction approach - Update metadata field names to reflect extraction rather than relevance checking * feat(benchmark): add selective file deletion option to clean_sample_outputs - Add --filename argument to delete only specific root-level files - Modify iter_sample_targets function to accept optional filenames filter - Implement validation for root-level filename constraints - Update function calls to pass filenames parameter - Add example usage for selective file deletion in documentation * feat(benchmark): add error count metrics to golden check statistics - Added golden_bad, session_bad, and both_bad calculation fields - Updated console output format to include error counts per question type - Modified table display to show both accuracy rates and error numbers - Enhanced statistical summary with additional error breakdown metrics * test(search): update search step tests with include_source parameter - Added include_source=False parameter to VectorSearchStep initialization - Added include_source=False parameter to Bm25SearchStep initialization - Maintained existing RuntimeContext parameters for both search steps - Updated test calls to match new constructor signature with include_source option	2026-07-13 21:26:27 +08:00
jinliyl	b1c9bf67bf	fix(embedding): make input truncation CJK-aware (#337) ... * fix(embedding): make input truncation CJK-aware * test(embedding): cover CJK-aware truncation budget	2026-07-13 17:43:48 +09:00
Ziyang Guo	e41b1673ad	fix(search): honor min_score in plain search steps (#338)	2026-07-13 16:32:16 +08:00
Ziyang Guo	c5eefe4da3	fix(search): expose markdown frontmatter on chunks (#314) ... * fix(search): expose markdown frontmatter on chunks * style(search): apply pre-commit formatting * fix(search): make frontmatter chunk metadata opt-in * fixup! fix(search): expose markdown frontmatter on chunks * feat(markdown): add include_frontmatter_keys_in_metadata allow-list opt-in --------- Co-authored-by: RerankerGuo <1875366113@qq.com> Co-authored-by: Ziyang Guo <121015044+RunMarshal@users.noreply.github.com>	2026-07-08 17:59:55 +09:00
jinliyl	2612d25959	feat(lme): add cli execution and agentic search tooling (#334) ... * feat(service): add CLI service for local job execution - Introduce CliService to execute single jobs locally without serving ports - Add prepare_start_config and should_precheck_start functions for CLI job setup - Update reme start command to use CLI service when job argument is provided - Change default service backend from http to cli in jinli_lme config - Modify SearchStep to use constants and rename configuration parameters - Add unit tests for CLI service functionality and configuration handling - Update file extension support to include json format in addition to md and jsonl * feat(search): add BM25 and vector search steps with configuration updates - Add Bm25SearchStep and VectorSearchStep classes with tool context deduplication - Register new search step components in index module - Update configuration to use separate vector_search and bm25_search endpoints - Modify LLM models from qwen3.7-plus/glm-5.1 to glm-5.2 variants - Adjust search parameters and remove hybrid search implementation - Configure embedding store as default in storage settings - Remove auto-memory and file catalog configurations - Update watch directories from multiple paths to session_dir only * feat(agent): add tool result offloading and workspace management - Add tool_results_dir configuration option for offloaded tool results storage - Implement ToolResultOffloadMiddleware to persist large tool results to files - Create WorkspaceBackend to standardize file operations across tools - Add configurable builtin tools selection with sequential execution option - Integrate middleware support for agent wrapper with offloading capability - Update application initialization to create tool results directory - Add safety mechanisms for filesystem operations with sanitized filenames - Enhance agent wrapper with configurable working directory handling - Upgrade agentscope dependency to version 2.0.4 for improved features # Conflicts: # reme/application.py * feat(benchmark): add LongMemEval agentic search and result management - Introduce AgenticAnswerStep for agent-based history search - Add LmePrepareJudgeStep and LmeSaveResultStep for evaluation pipeline - Implement AddDraftStep and ReadAllDraftStep for evidence accumulation - Update configuration with new agent wrapper and search parameters - Add comparison script for analyzing agent run differences - Include documentation for LongMemEval failure analysis - Enhance tool result offloading with skip options - Modify search defaults and indexing behavior * feat(agent): implement tool result offloading with system reminders - Added tool_result_offload_message parameter to agent wrapper reply method - Implemented configurable reminder template for offloaded tool results - Created system reminder messages when tool results are offloaded to files - Added Chinese user message template for agentic answer step - Updated tool result offloading middleware to use custom reminder templates - Enhanced agentic answer instructions to handle long tool results via draft storage * feat(scripts): add LongMemEval results summarization tool - Create summarize_lme_results.py script to analyze result JSON files - Implement command line interface with answer id and dataset root options - Add support for specifying index range with start and end parameters - Include option to show failure details and non-successful completions - Calculate completion statistics and accuracy metrics - Display detailed breakdown of yes/no/other judgements - Handle missing and unreadable result files gracefully - Format output with percentages and comprehensive summary statistics * feat(summarize_lme_results): add question type breakdown to result summary - Import defaultdict from collections module - Add by_type dictionary to track statistics by question type - Count completed, yes, no, and other responses for each question type - Display detailed breakdown table showing accuracy by question type - Include question type column when processing judgements - Print comprehensive summary with question type distribution - Calculate and display accuracy percentage for each question type category * feat(lme): switch to qwen3.7-max model and add shuffle functionality - Changed default LLM model from glm-5.1 to qwen3.7-max in jinli_lme.yaml - Added random module import for shuffle functionality - Implemented --shuffle argument with BooleanOptionalAction for dataset shuffling - Added --seed argument to control random seed for reproducible shuffling - Applied random shuffle to dataset indices when shuffle is enabled - Added console output showing shuffle operation and seed information * fix(cli): set default random seed for shuffle functionality - Changed default seed value from None to 42 for consistent shuffling behavior - Ensures reproducible results when using shuffle option without explicit seed - Maintains backward compatibility while providing deterministic defaults * refactor(benchmark): update agentic answer guidelines for grounding - Updated English instruction to emphasize strict grounding in retrieved context - Modified Chinese instruction to stress evidence-based responses without inference - Removed redundant conciseness requirement in both language versions - Enhanced clarity on proper use of draft saving and retrieval mechanisms - Strengthened emphasis against hallucination of unsupported facts * refactor(benchmark): update agentic search instructions and configuration - Replace separate vector_search and bm25_search with unified search tool - Update agent instructions to use single search tool with multiple strategies - Simplify Chinese instructions for search methodology - Add comprehensive search tool configuration with hybrid vector/BM25 capabilities - Increase model retry attempts from 1 to 3 for better reliability - Remove redundant tool references from job_tools list * feat(search): add configurable search limit with environment variable support - Remove hardcoded limit and min_score parameters from config schema - Increase LLM context size from 200000 to 1000000 - Add REME_SEARCH_LIMIT environment variable support for search configuration - Implement command line argument --search-limit to override default search limit - Add input validation to ensure search limit is positive - Modify subprocess execution to pass environment variables - Update search step to use dynamic default limit from environment or fallback to 5 * refactor(benchmark): remove agentic answer step and related configurations - Removed AgenticAnswerStep class and its registration - Deleted agentic_answer.yaml prompt configuration file - Removed agentic answer related job definitions from jinli_lme.yaml - Cleaned up tool result offloading middleware implementation - Removed tool_results_dir configuration field from application config - Deleted comparison and analysis scripts for agent runs - Removed agentic answer step from LME init module exports - Updated agent wrapper to remove tool result offloading functionality - Removed unused imports and dependencies in agent wrapper module * refactor(benchmark): remove unused LME result processing components - Removed LmePrepareJudgeStep and LmeSaveResultStep classes from benchmark module - Cleaned up imports and exports in lme module initialization - Removed unused middleware configuration from agent wrapper - Deleted obsolete result.py file containing deprecated result processing logic - Simplified agent instantiation by removing middleware parameter - Updated import statements to reflect removed dependencies * refactor(index): remove unused search steps and update imports - Remove Bm25SearchStep and VectorSearchStep from index steps module - Remove unused prepare_start_config and should_precheck_start exports - Move import statements to proper location in reme.py - Update test module to use direct import path for CliService - Remove vector_search and bm25_search configurations from jinli_lme.yaml - Add workspace directory environment variable configuration - Add docstring to getcwd method in agent wrapper - Remove empty middleware list from agent wrapper initialization * feat(index): add BM25 and vector search steps with tool context deduplication - Add Bm25SearchStep for plain BM25 keyword search with tool_context deduplication - Add VectorSearchStep for plain vector search with tool_context deduplication - Implement tool context state management with TTL-based deduplication - Add support for chunk deduplication across tool contexts within TTL window - Update index steps module to include new search step classes - Add test coverage for CLI metadata output functionality - Refactor CLI service to remove unused show_status parameter - Update documentation comments to reflect internal service configuration * feat(steps): add Python code execution capability - Introduce PythonExecuteStep to run Python code in subprocess - Add configuration for python_execute step in jinli_lme.yaml - Register python_execute in available tools list - Implement timeout handling with default 60 second limit - Capture stdout/stderr output and return code metadata - Add comprehensive unit tests for execution scenarios - Support workspace directory context for code execution - Handle timeout errors and runtime exceptions gracefully * refactor(python_execute): replace subprocess with asyncio for Python code execution - Replace subprocess.run with asyncio.create_subprocess_exec for non-blocking execution - Add _PythonResult dataclass to encapsulate execution results and timeout status - Implement proper timeout handling with asyncio.wait_for and process.kill() - Update metadata to include returncode and stderr when timeout occurs - Convert synchronous _run_python method to asynchronous implementation - Maintain backward compatibility while improving execution reliability * refactor(python_execute): replace subprocess with asyncio for Python code execution - Replace subprocess.run with asyncio.create_subprocess_exec for non-blocking execution - Add _PythonResult dataclass to encapsulate execution results and timeout status - Implement proper timeout handling with asyncio.wait_for and process.kill() - Update metadata to include returncode and stderr when timeout occurs - Convert synchronous _run_python method to asynchronous implementation - Maintain backward compatibility while improving execution reliability	2026-07-08 17:43:09 +09:00
xyf2020	82971ac5b0	feat(chunker): better json chunker and json chunker (#325) ... * feat(file_chunker): add dedicated JSON and JSONL file chunkers - Add JsonFileChunker: structure-aware chunking preserving nested key paths, optional list-to-dict conversion, size measured by json.dumps() char count - Add JsonlFileChunker: line-aligned sliding-window chunking with configurable overlap, supports char/byte mode switching - Register both chunkers in default.yaml (json for .json, jsonl for .jsonl) - Add comprehensive unit tests (21 + 20 test cases) * chore(config): update default chunker supported_extensions to txt/log * refactor(json_chunker): optimize _build_tree O(n²) serialization and rewrite tests - Fix O(n²) redundant json.dumps in _build_tree: * Empty containers handled directly as leaves (0 serialization) * Non-empty containers recurse first, then reconstruct+dump once * Only containers that become leaves pay serialization cost - Add _reconstruct_object/_reconstruct_array helpers - Remove dead code: _merge_json method - Apply user changes: min_element_size formula 0.01->0.05, threshold < to <= - Use indent=None for compact output (consistent with _SizeNode estimation) - Remove unused _text_size from JsonlFileChunker Test rewrite: - Replace try/finally boilerplate with make_json fixture - Group tests into TestXxx classes with pytest.mark.parametrize - Add TestOutputValidation: 9 parametrized scenarios verifying: * All chunks are valid JSON * Text length <= chunk_chars (with single-leaf tolerance) * Leaf-value concatenation matches original data (dict + array roots) - Add TestSizeNode: incremental size accuracy tests - Add TestDfsAlgorithm: path wrapping, DFS order, calibration tests - Update test_min_element_size_formula for new 0.05 multiplier - Update test_build_tree_structure for larger min_element_size * chore: apply black formatting to test files	2026-07-08 15:18:59 +08:00
jinliyl	eb471d7d94	fix(embedding): reject mismatched embedding dimensions (#330) ... * fix(embedding): enforce strict dimension matching for embeddings - Add _embedding_dim_matches method to validate embedding dimensions - Reject embeddings with mismatched dimensions instead of padding/truncating - Drop stale embeddings with wrong dimensions during loading and upsert operations - Disable embedding store when query dimensions don't match configured dimensions - Fail health checks when embedding dimensions don't match expected values - Skip chunks with wrong dimensions during FAISS index rebuild - Add comprehensive tests for dimension validation behavior * refactor(file_store): simplify conditional checks in vector search and test assertions - Combine multiple conditionals into single check for empty FAISS index - Replace explicit empty list comparison with boolean check for node embedding calls - Maintain same functional behavior while improving code readability * fix(embedding): harden dimension validation helpers	2026-07-08 14:23:38 +09:00