97 commits

Author	SHA1	Message	Date
jinliyl	99afc2604f	fix(release): harden embedding store and plugins for ReMe 0.4.1.9 (#503) ... * chore(release): prepare ReMe 0.4.1.9 * refactor(config): remove daily_cookbook and streamline plugin configs - Delete the entire daily_cookbook.yaml standalone application config - Remove qwenpaw dependencies verification and related CI workflow steps - Simplify release workflows by removing qwenpaw verification and enforcing reme-ai >=0.4.1.9 - Update plugin start commands and examples to use 'default' or 'demo' configs instead of daily_cookbook - Adjust imports and tests related to daily_cookbook removal and injected_job_kwargs enhancements - Refactor agent wrapper to support injected_job_kwargs for job parameter injection in auto-fin and daily-paper - Improve daily_paper digest prompt to include configured daily directory and correct historical search constraints - Update dependency versions in pyproject.toml files to require reme-ai >=0.4.1.9 and remove qwenpaw optional dependencies - Clean up unused environment variables and obsolete test cases related to daily_cookbook and verification steps * fix(local_embedding_store): retry batch computation on vector space changes - Add up to 3 attempts to recompute embedding batch if vector space changes during processing - Log warnings when maximum retries reached and discard stale results - Prevent caching results from outdated vector spaces to maintain consistency - Add tests to verify retry behavior and abort after continuous vector space churn fix(daily_paper): update digest search logic and tests - Change search to query existing memory, not only previous articles in daily_dir - Allow multiple searches outside daily_dir but limit links to dated markdown in daily_dir before today - Update test assertions to reflect revised search and linking rules * fix(embedding): retry vector space changes per request	2026-08-28 11:35:04 +08:00
jinliyl	3d2ecc60d2	feat(service): expose MCP through HTTP backend (#498) ... * feat(service): expose MCP through HTTP backend Serve JSON/SSE job endpoints and streamable HTTP MCP from one FastAPI application, sharing the same jobs and application lifecycle. Preserve the standalone MCP backend, add configurable MCP HTTP settings, update startup metadata and integration docs, and cover routing, lifecycle, configuration, and compatibility behavior with unit tests. * fix(service): preserve MCP request protections Route the exact MCP path through the complete FastMCP ASGI application so its middleware and state remain active. Reject non-literal MCP paths and validate reserved Job conflicts before tolerant service registration. Add regression coverage for middleware preservation, route syntax, and startup failure. * fix(service): reject encoded MCP paths Reject percent signs in mcp_path so ASGI path decoding cannot turn an accepted configuration into an unreachable route. Cover encoded slash, space, and double-encoded slash inputs.	2026-08-27 17:23:11 +08:00
jinliyl	513fb5b7f4	feat: extract Daily Paper into an independently packaged plugin (#491) ... * feat: extract Daily Paper into a plugin * fix: satisfy clean-environment quality checks * fix: address daily paper review feedback	2026-08-26 17:32:46 +08:00
jinliyl	efcc2b34d1	feat: simplify plugin setup and add management CLI (#485) ... * feat: simplify plugin setup and add management CLI * fix: isolate plugin CLI import side effects * refactor: streamline plugin validation * fix: route plugin CLI arguments independently * fix: support standard plugin source layouts	2026-08-23 17:58:20 +08:00
jinliyl	2f5fd46b44	refactor: deduplicate entry-point loading (#460) ... * refactor: deduplicate entry-point loading * fix: preserve config conflict error priority	2026-08-19 19:57:53 +08:00
jinliyl	618e8cec66	feat: add entry-point plugin system and extract Auto Fin (#459) ... * feat: add entry-point plugin system * fix: harden plugin config and client loading * docs(workflow): add detailed manual for publishing reme-auto-fin to PyPI - Provide step-by-step instructions for updating project.version and merging branches - Explain dependency verification for reme-ai on PyPI during build - Specify requirements for GitHub Actions secret configuration and version uniqueness - Describe manual workflow triggering and input of version number - Recommend publishing order for related projects - Clarify that only manual dispatch triggers publishing, no automatic triggers on push or tag * feat: support plugin-defined component types * refactor: simplify plugin configuration * fix: isolate plugin loading and defer client fallback * refactor: freeze built-in component registry * fix: isolate config entry point loading * fix: complete auto-fin package metadata	2026-08-19 17:23:23 +08:00
jinliyl	fd2894f939	fix: harden the 0.4.1.7 release configuration (#456) ... * fix(packaging): harden the Studio release workflow * chore(daily-paper): tune scheduled discovery defaults * fix(docs): link the ReMe blog to GitHub Pages * fix(docs): increase Chinese hero title spacing * refactor(docs): share hero title line spacing * fix(docs): keep desktop hero copy on two lines * fix(docs): widen the home hero description * style(docs): loosen hero title line height * fix(docs): hide Markdown frontmatter in rendered pages * docs(readme): simplify installation and remove standalone ReMe Studio instructions - Remove references to separate ReMe Studio package and static build steps - Clarify that `core` extra includes common integrations including Studio - Update installation instructions to use `pip install -e ".[core]"` - Remove detailed Studio usage and frontend development instructions - Note that Studio is included with `core` and optional via `web` extra - Simplify Quick Start guide by removing Studio usage step - Remove mentions of serving Studio with HTTP service when using extras - Update both English and Chinese README files accordingly * docs(readme): streamline and clarify memory design and operations - Remove redundant explanations about core extra installation - Simplify memory processing flow description for clarity - Clarify memory workspace directory default and customization - Condense automatic memory flow to emphasize rebuildable metadata - Refine search functionality explanation with RRF fusion details - Shorten and clarify agent integration description, removing redundancy - Update and simplify the operations command list, removing less common commands - Revise community and support section for conciseness and clarity - Maintain parallel updates in both English and Chinese README files * test(bump_version): add tests for version bumping and consistency checks - Add dynamic loading of bump_version and package_studio scripts for testing - Test that studio package and dependencies have matching versions - Implement fixtures to write temporary version files for testing - Add test ensuring bump_version updates all relevant files and dependencies - Add test to reject inconsistent version sources before writing - Refactor tests to use common REPOSITORY path variable - Include imports and setup for pytest in test file feat(bump_version): create script to update ReMe and Studio versions - Implement version reading from __init__.py and pyproject.toml files - Validate current versions are consistent across files before updating - Update version strings atomically to avoid partial writes - Ensure exact pinning of studio dependency in main package extras - Validate new version format against a safe pattern - Provide CLI interface to bump versions from command line - Raise errors if expected version declarations or pins are missing or duplicated * fix(release): validate split package publishing * fix(release): improve validation diagnostics * fix(release): sync docs and workflow inputs * fix(release): split PyPI publish jobs	2026-08-13 17:22:00 +08:00
jinliyl	215c1f72f2	feat: refine local-first research and memory workflows (#444) ... * feat: refine local-first research workflows * fix: delegate structured output tool choice * refactor(auto-fin): fetch and filter rolling CLS news * fix(auto-fin): keep imports portable across platforms * feat(auto-fin): expose CLS fetch controls * fix(auto-fin): propagate configurable news window * feat(auto_fin): normalize hybrid wikilinks in report body - Add _normalize_hybrid_wikilinks method to remove redundant Markdown destinations - Use regex to identify hybrid wikilinks with optional destinations - Replace redundant destinations with simpler wikilink format for clarity - Ensure normalization is failure-safe with exception handling and logging - Update report body normalization process to apply hybrid wikilink fix - Add unit tests to verify correct normalization and failure safety behavior * fix(dream): serialize integration with application-wide asyncio lock - Add application-wide asyncio.Lock to serialize digest writes during integration - Update _snapshot_digest to capture metadata per bucket - Validate bucket association when recovering from file changes - Add tests ensuring recovery only from the correct bucket - Add tests confirming integration lock is shared across application context - Enhance strict topic YAML loading validation in dream utils - Add tests for strict topic loading rejecting invalid or lossy fields * fix(cookbook): enable configurable job_tools for digest and merge steps - Update daily_cookbook.yaml to add job_tools: [memory_search, read] in digest steps - Modify DailyPaperDigestStep to read job_tools from kwargs instead of fixed list - Modify AutoFinMergeStep to similarly read job_tools from kwargs - Update tests to pass job_tools explicitly when invoking these steps - Remove hardcoded _TOOLS constants and replace with dynamic job_tools handling * fix: retry incomplete dream receipts * perf(pdf): increase max PDF pages limit from 20 to 35 - Updated configuration max_pdf_pages from 20 to 35 in daily_cookbook.yaml - Modified code to extract up to 35 pages instead of 20 in analyze.py - Updated README and README_ZH to document the increased max_pdf_pages - Adjusted unit test assertions to reflect new max_pdf_pages limit of 35 * fix memory integration and daily paper links * docs clarify cookbook tool usage	2026-08-11 23:32:34 +08:00
jinliyl	9533c17d51	feat(web): serve workspace from HTTP service (#446) ... * feat(web): add the ReMe workspace frontend * feat(web): serve workspace from HTTP service * test(web): satisfy pylint docstring checks * fix(web): use same-origin API safely * fix(web): preserve API route semantics	2026-08-11 23:32:24 +08:00
jinliyl	072cb6a55b	feat(daily-paper): add opt-in Hugging Face mirror support (#437) ... * feat(daily-paper): add Hugging Face mirror switch * refactor(daily-paper): simplify the HF mirror switch and warn on ignored env The switch was a three-state bool|None where None preserved the legacy environment-driven selection, but no production caller ever passes None -- collect.py always resolves an explicit bool. Collapse it to a plain bool defaulting to False. HF_MIRROR_URL no longer redirects traffic on its own, so warn when it is configured while the mirror stays disabled; a mirror-only setup would otherwise fall back to the official site with no signal. Both READMEs now record the behavior change and stop presenting the two mirror variables as symmetric -- arXiv remains environment-driven while Hugging Face is gated on the job parameter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(daily-paper): address mirror configuration feedback --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>	2026-08-10 15:57:24 +08:00
jinliyl	d5e0d2837b	refactor: rebuild auto-fin and daily-paper cookbooks on structured-output agents (#432) ... * refactor: rebuild auto-fin and daily-paper cookbooks on structured-output agents Rework the auto-fin and daily-paper cookbooks to run on structured-output LLM agents instead of Claude Code agent wrappers, replace the SSH proxy with data-source mirrors, and rewrite the affected unit tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(auto_fin): unify JSON output serialization and writing - Extracted _write_output static method to serialize and write Pydantic models as compact JSON - Replaced inline JSON dump and write calls with _write_output usage across auto_fin steps - Added _report_path and _current_report for managing intra-day reports in AutoFinMergeStep - Updated auto_fin merge step to write output via new _write_output method - Enhanced news reading with caching in AutoFinHistoryStep - Refined returns calculation to handle events before close on non-trading days correctly feat(daily_paper): improve note path resolution and metadata handling - Introduced iter_note_metadata generator for safe Markdown frontmatter iteration - Added resolve_unique_note_path to avoid note filename conflicts on disk and in used titles - Updated analyze, collect, digest, and select steps to use centralized constants and helpers - Used utc_now_iso for consistent timestamping in metadata - Replaced direct frontmatter loads with iter_note_metadata in collect and analyze steps - Replaced hardcoded paper selection count with PAPER_COUNT constant in all relevant places - Added _MAX_SELECT_ATTEMPTS constant in select step for attempt management - Improved error messages for filename validation in daily paper title normalization feat(auto_fin): add multi-run cron schedules for intraday refinement - Defined three auto_fin cron jobs at 09:30, 11:30, and 18:00 Shanghai time for gradual report updates - Each intraday run adds evidence cumulatively instead of replacing prior output wholly - Updated daily_cookbook.yaml to register new cron schedules and remove legacy 12:00 cron refactor(auto_fin_data): clean ETF code handling and page limits - Replaced hardcoded DEFAULT_ETF_CODES with required non-empty config value "etf_codes" - Added constants for major news and fund page limits to control pagination - Improved ETF name extraction logic to handle missing fields consistently fix(auto_fin_merge): fix report retrieval and merging logic - Added support for getting current intra-day report in addition to previous day's report - Modified merge template to include prior and current report sections for better context - Adjusted report path handling to consistently use Path objects test(auto_fin): add coverage for returns calculation and report retrieval - Added test for returns when event occurs before close on non-trading day, checking next session entry - Added test for previous and current report retrieval feeding merge context with disk files - Extended test asserts for auto_fin cron schedule changes in config style(daily_paper): reorder and cleanup imports - Reorganized imports in _common.py for clarity and added missing collections.abc.Iterator import - Cleaned up commented and unused imports across daily_paper steps * feat: add configurable upstream mirror proxy * style: format auto-fin data step * fix: align cookbook mirrors and contracts --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-08-07 23:53:14 +08:00
jinliyl	e05b201da9	feat(backend): improve workspace support for web clients (#420) ... * feat(backend): improve workspace support for web clients * fix(config): preserve the default workspace directory * chore(reme): bump version to 0.4.1.5 - Update __version__ from 0.4.1.4 to 0.4.1.5 in initialization file * fix(chat): disable builtin tools in read-only mode * fix(agent): make builtin tools opt-in * fix(list): tolerate files removed during mtime sort * fix(chat): expose complete read-only job set	2026-08-07 23:52:56 +08:00
jinliyl	9218a2d0e3	refactor: derive dialog paths from session_dir (#421) ... * refactor: derive dialog paths from session directory * fix: normalize configured session paths * fix: align dialog watch paths with writers * fix: reject absolute session directories	2026-08-06 17:07:14 +08:00
jinliyl	f31daf1949	Revert "feat(backend): improve workspace support for web clients (#417)" (#419) ... This reverts commit b00eb0a9ea.	2026-08-05 23:17:08 +08:00
jinliyl	b00eb0a9ea	feat(backend): improve workspace support for web clients (#417)	2026-08-05 23:07:05 +08:00
xyf2020	5bc46c88b6	feat(benchmark): enhance session memory retrieval and isolate benchmark assets (#409) ... * chore(benchmark): isolate dataset/workspaces/results per benchmark - Move shared benchmark/{datasets,memory_workspaces,results} into per-benchmark subdirs benchmark/<name>/{dataset,workspaces,results} - Update beam/longmemeval config.yaml and run.py path defaults - Relocate longmemeval download.py to benchmark/longmemeval/ (downloads into dataset/ subdir); inline dataset download docs into README - Update .gitignore: benchmark/*/{dataset,workspaces,results}/ - Move result-{beam,longmemeval}.md to benchmark/results_md/ and drop result- prefix; update README links - Fix stale path refs in llm_judge.py and logs/demo_search_format.py * feat(benchmark): add read tool to agentic answer and update BEAM results - Add 'read' to job_tools in BaseAgenticAnswerStep for file reading capability - Document read tool usage in lme/agentic_answer.yaml system prompt - Update result-beam.md with latest evaluation scores (OVERALL: 0.623/0.580) * feat(auto_memory): add source line-number markers for note traceability - Add _format_history hook in AutoMemoryStep with line-number annotation - Override in BeamAutoMemoryStep to prefix each turn with [Ln] for citation - Add session_file variable to prompt templates for source marker paths - Simplify repeated extraction rules by referencing system prompt - Enhance agentic_answer search strategy (multi-search, read tool hint) - Add warning log on ReadStep failure * feat(beam): enhance auto_memory with source markers and pilot ingest tooling * refactor(beam): rename max_chunk_words to max_segment_words, drop one-off pilot scripts * feat: add CompressorStep and search_v2 dual-mode session compression - Add CompressorStep (reme/steps/evolve/compressor.py) for direct LLM text compression with optional query-guided relevance filtering - Extend search_v2_step to support query-aware and query-independent session transcript compression via _compress injected kwargs - Refactor _source_format.py: split into render_chunk_entries + join_chunk_entries; session chunks now render line-aligned with L<n>: prefixes for verbatim/compressed parity - Add JOB_TOOLS and INJECTED_JOB_KWARGS to BaseAgenticAnswerStep for per-subclass tool and parameter injection - LmeAgenticAnswerStep injects _search._compress payload to enable query-aware compression during benchmark evaluation - Record compression ablation results in result-longmemeval.md - Add unit tests for CompressorStep and search compression paths * refactor(compress): relax session compression to lenient format-preserving strategy and update LME results * refactor(benchmark): make session compression config-driven via compress_session flag Move session-transcript compression from LME hard-coded injection to a runtime context flag set by evaluation.compress_session in each benchmark config. Compression is off by default for both BEAM and LME, and BaseAgenticAnswerStep now conditionally injects the _search compress payload only when the flag is truthy. * feat(lme/auto_memory): add source attribution markers with line numbers Add _format_history to annotate each turn with [Ln] line numbers and expose {session_file} in prompts so the agent can emit bare wikilink-style source markers like [[session/dialog/s1.jsonl#L1-L2,L5-L6]] at the end of factual entries. Consolidate the per-prompt body/format rules into references to the system prompt to avoid drift, and add frontmatter- protection guidance for the edit tool. * feat: improve agentic answer prompt and update beam 100K results - Strengthen abstention rule: prohibit extrapolation from related but non-direct evidence - Add multi-angle search after preliminary answer to check for conflicting/supplementary/updated information - Add max-iteration fallback to 'Information not found' - Update beam.md with 100K results (agentscope 2.0.4.post1, from scratch) including per-type token consumption and memory construction stats - config.yaml: 100K dataset, 20 workers for BEAM evaluation - run.py: add memory construction token usage tracking (default agent) - Overall: 0.635 → 0.654 (+0.019), contradiction_resolution: 0.338 → 0.478 (+0.140), abstention: 0.500 → 0.525 (+0.025) * feat(read): add session-aware formatting for read tool and update BEAM eval - Add truncate_session_output in _file_io.py to render jsonl session lines as [speaker @ time] content before byte-budget truncation - Add read_step_format_session flag to ReadStep, honoring injected job kwargs (precedence) and YAML fallback - Inject read_step_format_session=True into BaseAgenticAnswerStep so agentic answer reads render session transcripts human-readably - Refine BEAM agentic_answer prompt: continue multi-angle search after preliminary answer, forbid fabrication/extrapolation - Update BEAM config to 1M variant and add sequential 100K-eval / 1M-build shell script - Refresh benchmark/results_md/beam.md with latest results * chore(config): disable expand_links in beam and lme search_v2 configs * refactor(beam): drop one-off sequential 100K-eval-then-1M-build script * fix(benchmark): add compressor job to beam config and fix BEAM clone instructions - Add compressor job and compressor as_llm component to reme/config/beam.yaml (aligned with lme.yaml) so that compress_session: true works for BEAM - Add graceful degradation guard in search_v2._compress_session_entries: when the compressor job is missing from the active config, log a warning and skip compression instead of raising 'Job compressor not found'. Skipped when there is no app_context so unit tests mocking run_job still drive compression behavior. - Fix BEAM download instructions in README.md/README_ZH.md: add mkdir -p before cd benchmark/beam/dataset (the directory is gitignored and absent in a fresh clone) * fix(steps): guard compressor exceptions and fix ReadStep boolean override 1. search_v2: catch per-entry exceptions from run_job('compressor') inside compress() so asyncio.gather never propagates a compressor failure (e.g. temporary LLM outage). The failing entry keeps its original body while remaining entries are still compressed, preserving already-retrieved search results. 2. read: replace 'context_value or yaml_value' with an existence check so that a runtime-injected False can explicitly disable a YAML-true read_step_format_session flag. Add focused unit tests for both paths. * fix(search_v2): use existence check for strict_date_filter boolean override Replace 'context_value or yaml_value' with an existence-based check so that a runtime-injected False can explicitly disable a YAML-true strict_date_filter flag, consistent with the read_step_format_session fix. * refactor(search): simplify strict_date_filter fallback to truthiness-or * style(test): rename unused param to satisfy pylint W0613 --------- Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>	2026-08-05 19:23:42 +08:00
jinliyl	e256c556ca	feat: add workspace web APIs and star growth report (#416)	2026-08-05 18:03:37 +08:00
jinliyl	eac8223387	feat: add frontend-ready wikilink graph APIs (#414)	2026-08-05 16:45:50 +08:00
Sen Huang	550317c3bf	Revert "feat(plugin): add ReMe integration for Codex (#372)" (#400) ... This reverts commit a367c2ce13.	2026-07-29 18:14:05 +08:00
DiegoCluv7	a367c2ce13	feat(plugin): add ReMe integration for Codex (#372) ... * feat(plugin): add ReMe integration for Codex * fix(plugin): fix Codex plugin port, transcript ingestion, and Windows support * fix(plugin): correct Codex transcript schema, path validation, and hook fixes * test(plugin): add MCP round-trip tests * fix(plugin): rewrite parser and tests. * fix(plugin): reserve id-less messages, cover marketplace manifest, error handling, path fixes, and main sync Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-07-29 18:11:29 +08:00
xyf2020	4eb2adf961	feat(faiss_file_store): upgrade FAISS to HNSW index with async reindex (#390) ... * feat(file_store): upgrade FAISS to HNSW index with async reindex and path constraint - Replace IndexFlatIP with IndexHNSWFlat for better recall/speed tradeoff - Add dynamic efSearch (limit * 5) scaled to query request size - Add async_reindex option: background rebuild with generation-based invalidation - Extract _delete_nodes() in LocalFileStore for subclass reuse - Add unit tests for file store consistency * fix: resolve pylint warnings in faiss store and test file * refactor(file_store): replace generation-based reindex with event-flag worker - Replace _reindex_generation/lock/task with a single long-lived worker coroutine consuming an asyncio.Event flag; repeated submissions coalesce - Use local index reference in vector_search to avoid TOCTOU on self._faiss_index - Pass index explicitly to _set_ef_search for consistency - Track _index_writes to re-arm reindex after concurrent writes - Update tests to match new internal API * fix: resolve pylint too-many-return-statements and implicit-booleaness warnings * feat(file_store): add refine maintenance hook and incremental embedding backfill - Add refine() idle-time maintenance hook to BaseFileStore/LocalFileStore - FaissLocalFileStore: incremental vector add on backfill instead of full rebuild - Dynamic tombstone compaction threshold scaled by index size - Add RefineStoreStep with daily cron job (refine_store_cron) - Enable faiss backend and embedding_store by default in default.yaml - Add unit tests for faiss index maintenance * chore(deps): promote faiss-cpu to core dependencies faiss backend is now the default file_store, so faiss-cpu moves from the optional [core] extra to the base dependencies list. * feat: rename refine_store to optimize_index and add vecdb_path_constraint - Rename refine_store step to optimize_index with cron job scheduling - Add vecdb_path_constraint to file_store components - Update default.yaml with optimize_index_cron and faiss backend comment - Update memory_search docs (en/zh) for FAISS vector management - Update unit tests for index maintenance * feat(faiss): add embedding digest to reject stale sidecar after partial dump Add _chunks_embedding_digest() that computes an order-independent SHA-256 over (chunk_id, float16 embedding) pairs. The digest is written into the idmap sidecar at dump time and verified at load time. A mismatch means the sidecar vectors belong to a different chunk generation than the authoritative JSONL — detectable even when the live-ID set is unchanged (same-ID in-place update crash window). Add test_faiss_rejects_stale_sidecar_after_partial_dump reproducing the crash-between-writes scenario and asserting digest-based rejection. Compress verbose docstrings/comments in existing tests for pylint line budget. --------- Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>	2026-07-27 19:54:38 +08:00
jinliyl	1687179f84	feat: add Auto Fin cookbook and managed outbound proxy support (#392) ... * feat: add ssh proxy * feat: add ssh proxy * feat: add ssh proxy * feat: add ssh proxy * feat: add prompt * feat: add agent wrapper * feat: add agent wrapper * feat: add agent wrapper * feat: add tushare skill * feat: add tushare skill * feat: add tushare skill * feat: add none stream * chore(deps): update dependency versions in pyproject.toml - Bump claude-agent-sdk from 0.2.123 to 0.2.126 - Upgrade pre-commit to version 4.6.1 or higher - Upgrade pytest to version 9.1.1 or higher * feat(agent_wrapper): add session compaction support and unify session commands - Introduce compact_session method to BaseAgentWrapper and implement it in AsAgentWrapper, CcAgentWrapper, and CodexAgentWrapper - Add session_command module with SessionCommandResult dataclass and handle_session_command function for /clear and /compact commands - Update __init__.py exports to include session_command handlers - Modify DingTalkWaitStep to handle session commands via handle_session_command function - Remove streaming mode from DingTalkWaitStep and simplify reply handling to final Markdown replies only - Add unit tests for session compaction methods and session command handling across wrappers and DingTalk integration - Clean up and remove obsolete streaming and card rendering code from DingTalk wait step - Adjust daily_cookbook.yaml to remove stream and card_update_interval config entries for DingTalk wait step * feat(auto_fin): add Auto Fin simulated portfolio cookbook workflow - Add comprehensive Auto Fin schema exports for multiple models and enums - Implement base class and helpers for Auto Fin analysis steps - Create file, state, and formatting utilities for Auto Fin with atomic file writes and locking - Define Auto Fin pipeline with four analysis agents: backtest, event, portfolio, and US correlation - Register Auto Fin package in cookbook workflows and schema initialization - Add detailed documentation in markdown describing the system design, workflow, and data contracts * feat(outbound_proxy): add application-scoped outbound HTTP proxy components - Introduce BaseOutboundProxy and OutboundProxyEndpoint as core contracts - Implement FixedHttpOutboundProxy for external HTTP proxy integration - Add SshHttpOutboundProxy providing SSH-backed local HTTP proxy tunnels - Register outbound proxy components in component registry and enumeration - Update components package to include outbound_proxy module - Add dependency on pproxy for SSH HTTP proxy bridging - Include comprehensive unit tests covering proxy lifecycle, validation, environment merging, error handling, readiness, and monitoring mechanisms * refactor(network): replace SSH proxy with explicit HTTP outbound proxy - Remove SSH proxy helper implementation and references in codebase - Add support for explicit HTTP proxy URL in arXiv and HuggingFace clients - Modify clients to use async context manager for consistent resource handling - Update daily paper steps to forward outbound proxy configuration explicitly - Change tests to cover new proxy usage model and remove SSH proxy mocks - Add outbound proxy component configuration in daily_cookbook.yaml - Ensure proxy URL usage disables environment trust in HTTP clients - Fix app context component enum access to be defensive against missing keys * feat(agent_wrapper): add managed proxy support for command environments - Introduce BaseOutboundProxy binding in BaseAgentWrapper for outbound proxy management - Add bash_environment and command_proxy_environment properties to apply proxy settings - Update WorkspaceBackend instantiation in AsAgentWrapper to use bash_environment - Inject managed proxy export commands into Claude Code Bash commands via hooks - Enhance CodexAgentWrapper to include managed proxy in shell environment policy - Modify daily_cookbook.yaml steps to specify outbound_proxy as default where needed - Add comprehensive unit tests verifying managed proxy injection and environment isolation - Ensure subprocess_environment remains unchanged while proxy is applied selectively to commands * refactor(memory): replace search job_tools with memory in daily cookbook config - Change workspace_dir default from .reme to reme_workspace - Replace search job_tools with memory across multiple components and jobs - Update descriptions to reflect long-term memory retrieval instead of search - Modify system prompts to instruct using memory for retrieving notes - Adjust unit tests to verify memory job_tools and job presence instead of search - Ensure consistency in configuration and tests for memory backend usage * refactor(config): rename memory to memory_search in daily cookbook config - Change all occurrences of "memory" to "memory_search" in job_tools and job definitions - Update related system prompts to reflect the new memory_search terminology - Modify unit tests to assert the presence of memory_search instead of memory - Ensure consistency across skills, job tools, and backend configurations in multiple components * feat(auto_fin): add deterministic quantitative research and ranking fusion - Introduce new schema models: EtfScore, RankingMetrics, ExtremeAnalysis, DimensionRanking, and FusionRanking to represent deterministic research outputs - Add ranking data to event, backtest, us_correlation, and portfolio analysis outputs - Implement ranking_section renderer to format Top20 scores and diagnostics in Markdown - Develop AutoFinQuantStep for deterministic ETF ranking using TuShare data, Polars, and a custom extremely randomized tree ensemble - Integrate quantitative rankings into backtest and portfolio analysis steps and reports - Extend auto_fin pipeline with new quant_enabled and quant_required config options - Enforce ranking constraints like unique codes, contiguous ranks, and normalized fusion weights - Update analysis YAMLs with rules limiting data freshness, universe, and ranking usage - Incorporate ranking outputs into all major markdown report bodies in Auto Fin pipeline - Add concurrency-limited asynchronous TuShare client to fetch required market data - Introduce cross-sectional rank correlation and NDCG metrics for ranking quality evaluation * feat(auto_fin): implement stage-wise notification and reporting for analysis pipeline - Refactor notification config in daily_cookbook.yaml to support dispatch steps - Update AutoFinNotificationStep to deduplicate notifications per run stage - Add _notify_stage method in pipeline to send notifications for each analysis stage - Implement persistence and notification for event, backtest, US correlation, and portfolio stages - Modify pipeline flow to persist reports and notify after each stage completion - Adjust metadata to track notifications and errors per stage - Update tests to verify stage-wise notification sending and deduplication - Remove older combined report persistence in favor of modular stage handling * feat(auto_fin): add outbound proxy support for Tushare API usage - Introduce BaseOutboundProxy reference in AutoFinPipelineStep and AutoFinQuantStep - Update TushareResearchClient and trade calendar fetch to accept and use proxy URL - Create _ProxiedTushareApi adapter to route Tushare requests via explicit HTTP proxy - Modify create_tushare_api utility to optionally return proxied API client - Add unit tests covering proxy forwarding and client behavior with managed proxies - Ensure proxy usage respects explicit proxy URL over environment fallback - Integrate outbound proxy into data fetching and quantitative research steps * feat(auto_fin): enforce checkpoint time validation and add state models - Introduce AnalysisState base class and specific states for event, backtest, and US correlation analyses - Replace analysis output types with corresponding state classes in run schemas - Add require_checkpoint_reached method to validate decision_at/data_cutoff against current time - Enforce checkpoint time checks before analysis steps in event, backtest, portfolio, and quant analyses - Refactor quant data loading to include adjustment factors and apply price adjustments without fallback - Update analysis YAML docs to require real-time checkpoint validation and forbid using future data - Improve portfolio run serialization by excluding redundant legacy fields and nested proposed actions - Add helper to extract readable sections from persisted checkpoint documents - Fix event analysis output validation to reject events and sources with future timestamps * feat(auto_fin): auto-select latest reached checkpoint if none specified - Extend checkpoint config to accept empty string for auto selection - Add static method to compute latest checkpoint reached by current time - Modify pipeline step to auto-select checkpoint based on trade calendar and time - Adjust force flag default depending on whether checkpoint is explicit or auto - Log details when checkpoint is auto-selected to improve observability - Add comprehensive tests for auto checkpoint selection logic and edge cases - Remove deprecated default and required constraints from force parameter in config * refactor(auto_fin): unify datetime comparison with compare_datetimes utility - Replace direct datetime comparisons with compare_datetimes function calls - Use cmp_to_key with compare_datetimes for sorting datetime tuples and lists - Update validation logic in backtest, event, analysis, and ledger modules for consistent datetime handling - Add unit tests to verify handling of naive and aware datetime comparisons in event and backtest validations - Ensure marked_at and interval_end timestamps are set and compared consistently using compare_datetimes - Improve correctness of ordering and conditional checks related to timestamps throughout auto_fin steps and ledger code * feat(auto_fin): add datetime comparison helper for mixed timezone data - Implement compare_datetimes function to handle naive and aware datetimes - Ensure naive datetime is interpreted in the known timezone of the counterpart - Facilitate comparisons between legacy and timezone-aware Auto Fin data - Add module docstring explaining purpose of the helpers * docs(auto_fin): enforce unique ETF representative per sub-theme in analysis rules - Update backtest.yaml to recommend or highlight only one ETF per sub-theme for ETF analyses - Modify event.yaml to map only one representative ETF per sub-theme, avoiding duplicate recommendations - Revise portfolio.yaml to restrict holdings/buys to a single ETF per sub-theme, preventing repeated buys of highly overlapping ETFs - Adjust us_correlation.yaml to retain only one representative A-share ETF per sub-theme for mapping or recommendation - Add test to verify presence of new sub-theme uniqueness guidance in step prompts * feat(auto_fin): separate draft model and include deterministic fusion ranking - Introduce _PortfolioProposalDraft pydantic model for agent-authored fields before ranking - Discard any "fusion_ranking" data from draft to prevent conflicts with canonical ranking - Modify AutoFinPortfolioStep to receive draft, enrich with fusion_ranking, and produce final output - Update tests to use _PortfolioProposalDraft and validate deterministic fusion ranking propagation - Add async test verifying fusion ranking is correctly set in portfolio output with no errors * refactor(auto_fin): rewrite and simplify Auto Fin schema and steps - Remove legacy Auto Fin analysis step modules and helpers - Replace complex ranking and portfolio models with simplified current-news models - Update schema to focus on news-case workflow with new domain models - Remove A-share decision checkpoints and backtest details from schema - Simplify recommendation and decision output structures - Clean up deprecated state and utility functions - Update Auto Fin steps initialization to new pipeline steps only - Improve uniqueness validation for themes and ETFs in research plan * feat(auto_fin): implement full local cache and analysis workflow for Auto Fin - Add AutoFinDataStep to prepare and cache daily TuShare data with lookback - Add AutoFinAnalysisStep to analyze cached data and generate Markdown report - Implement detailed time window, ETF filtering, and historical case validation - Introduce YAML prompts for planning and decision-making steps - Update .gitignore to include reme_workspace/ - Clean up config and import structure for auto_fin steps - Remove old pipeline.py and consolidate functionality into new modules - Use polars for efficient CSV reading and data processing - Ensure atomic writes and strict JSON serialization for cache files - Enforce rules on news timing, ETF universe, and historical case usage * fix(auto_fin): restrict news data source to '财联社' in analysis and cache - Update analysis templates to specify current news as from '财联社' only - Modify news fetching functions to filter by source '财联社' - Add validation method to check cached news source correctness - Update news caching logic to exclude non-'财联社' news - Enhance unit tests with multiple sources to ensure filtering works - Confirm news API calls include source filter parameter as '财联社' * refactor(auto_fin): convert I/O methods to asynchronous implementations - Change _news, _dataset, and _theme_data methods to async for improved concurrency - Move JSONL and CSV reading operations to asynchronous wrappers using asyncio.to_thread - Remove synchronous _read_jsonl and _read_csv functions, integrate them as static async class methods - Update cache validation methods to async, awaiting I/O operations accordingly - Adjust usage of dataset and news retrieval in analysis step to await asynchronous methods - Add async unit test to validate JSONL reading with unicode line separators - Preserve existing functionality while enabling non-blocking file and data access * fix(nx_file_graph): defer networkx import and improve dependency handling - Move networkx import inside NxFileGraph constructor for lazy loading - Raise ImportError with original exception context if networkx is missing - Remove module-level fallback assignment of nx to None - Expand test to block loading of multiple optional core dependencies eagerly - Change exception type in test from ModuleNotFoundError to AssertionError - Update test comments to reflect broader optional dependency checks * feat(embedding_store): add quota retry delay mechanism for embedding requests - Introduce quota_retry_delay parameter to configure wait time before retry on quota exhaustion - Implement detection of insufficient quota errors in LocalEmbeddingStore without external SDK - Add retry logic with custom delay when quota is insufficient during embedding requests - Update configuration to set max_retries and quota_retry_delay defaults for embedding store - Add unit tests covering quota exhaustion retry behavior with delay and opt-in control - Ensure existing retry behavior remains unchanged if quota_retry_delay is not set * feat(auto_fin): add detailed logging to analysis and data fetching steps - Add _preview static method for bounded diagnostic output in analysis.py - Log prompt start, completion, errors, and validation details in _reply method - Add info logs for major processing steps in execute method of analysis.py - Add debug and info logs for cache validation, data fetching, and pagination in data.py - Log conditions for skipping reports and cache plans in data.py execute method - Log download summaries and cache writes for news and ETF data - Improve error logging with exception details in cache validation functions - Ensure all logs include context such as record counts, paths, and parameters * refactor(auto_fin): overhaul Auto Fin workflow and schema contracts - Replace old Auto Fin schema models with comprehensive new data classes - Remove legacy Auto Fin analysis step in favor of modular agent-based steps - Introduce AutoFinAgentStep for validating structured agent replies - Simplify data cleaning and JSONL writing utilities for news cache - Remove synchronous and asynchronous dataset methods from analysis step - Redefine Auto Fin analysis configuration for 360-day news retention and multi-step pipeline - Remove embedded analysis prompt templates and replace with agent-driven logic - Update __init__.py exports to match new step implementations and remove deprecated classes - Improve error handling and validation in agent step reply processing - Clean up redundant imports and unused code in analysis and data preparation modules * feat(auto_fin): add detailed logging for analysis and data processing steps - Add timing logs to measure agent prompt processing duration in analysis.py - Log news cache hits and news write paths with record counts in data.py - Include detailed info logs for news download start and completion in data.py - Add start, progress, and completion logs with topic and event counts in history.py - Log start and completion of merge step including path and ETF count in merge.py - Add start and done logs with window and news counts in topic.py * feat(auto_fin): enhance schema and steps with detailed ETF and event modeling - Replace and add multiple AutoFin schema classes to support detailed ETF selection, historical research, market analysis, forecast models, and report output with validation - Implement Shanghai timezone normalization and strict validation in schema models - Remove deprecated AutoFin analysis agent step and consolidate reply handling in base step - Introduce AutoFinStep base class with shared helpers for prompt handling, data fetching, logging, and JSONL file operations - Add AutoFinDataStep to manage daily news data complete with schedule validation, caching, and source validation logic - Update cookbook configuration to customize auto_fin step parameters and simplify outbound proxy settings - Refactor imports and clean unused code for better maintainability * feat(auto_fin): introduce detailed historical event resolution and market similarity analysis - Add AutoFinHistoricalEventReference and AutoFinHistoricalSimilarity models for refined event referencing and similarity judgment - Implement validation to ensure non-empty critical fields and uniqueness of historical news IDs - Develop method to resolve Agent-selected historical event references from workspace files with strict path and existence checks - Enrich historical events with market entry and future returns data after resolution - Redesign market step to calculate similarity-weighted ETF forecasts based on matched historical event similarities - Enforce validation on matched historical events for uniqueness and proper weight summation - Simplify merge step output to final Markdown report without YAML frontmatter and redundant fields - Update user instructions for history search, market, and merge steps to reflect new data structures and responsibilities - Adjust test suite to cover new schema and step behavior changes, including enhanced validation and JSON output formats * feat(auto_fin): add new cron jobs and output analysis jsonl - Add new cron jobs auto_fin_1145_cron and auto_fin_1800_cron with auto_fin_steps - Change auto_fin_0930_cron schedule to run Monday to Sunday - Extend merge step to write analysis data to auto_fin_analysis.jsonl - Update unit tests to verify new cron jobs and their steps configuration * fix(auto_fin): improve atomic file write and refresh daily index - Change temporary file naming to include UUID for uniqueness and hidden prefix - Replace atomic write method from using Path.replace to os.replace with safe unlink - Add import and use os.replace for safer file replace operation - Refresh daily index after writing auto finance markdown and JSONL files - Import and call refresh_day_index in merge step to update file index asynchronously * docs(cookbook): add optional SSH proxy configuration in README files - Introduce optional SSH proxy setup in auto-fin and daily_paper cookbooks - Provide instructions to enable outbound proxy via `daily_cookbook.yaml` and environment variables - Add `REME_PROXY_IP` and `REME_PROXY_ACCOUNT` environment variables descriptions in multiple README files - Update English and Chinese README and README_ZH documents with proxy details - Maintain consistent formatting of environment variable tables across documents * fix(file_io): include schema_version in hidden metadata keys - Added "schema_version" to _INDEX_HIDDEN_METADATA_KEYS in _daily_index.py - Updated _render_notes_block to always include additional keys regardless of schema_version fix(deps): move pproxy dependency to later in pyproject.toml - Removed pproxy from early dependencies list - Added pproxy back near the end of dependency list for better ordering fix(outbound_proxy): require pproxy package for ssh_http proxy - Added importlib.util check for pproxy package presence - Raise RuntimeError if pproxy is not installed when using SSH HTTP outbound proxy - Improved error message suggests installing reme-ai with 'core' extra * docs(readme): update News section with new Cookbook workflows - Clarify introduction of optional Cookbooks with Daily Paper and Auto Fin workflows - Update English README to reflect both paper discovery and file-native ETF event research - Revise Chinese README to include financial news and historical market data research capability - Maintain announcement of paper acceptance at Findings of ACL 2026 * feat(auto_fin): add calculation results to final Markdown output - Implement _calculation_results to summarize forecast for each ETF analyzed - Include program-calculated results in the JSON input for the Markdown report - Update YAML template to incorporate calculation results and adjust recommendation rules - Refine recommendation logic to rely on event impact judgments combined with calculation outputs - Modify tests to verify presence of calculation results and updated report content and format * up prompt * fix(keyword_index): ignore non-indexable chunks during keyword sync - Add is_indexable method to base and BM25 keyword index classes to check text tokenizability - Update local file store to exclude non-indexable chunks from expected document IDs to prevent rebuild - Fix JSONL chunker to correctly handle Unicode line separator U+2028 inside JSON strings without splitting - Add test to ensure non-empty but non-indexable chunk does not trigger keyword index rebuild - Add test to verify U+2028 character does not cause incorrect JSONL record splitting	2026-07-25 18:09:39 +08:00
jinliyl	46adb5ae1e	feat: add daily paper cookbook and DingTalk agent integration (#385) ... * feat(daily-paper): add daily paper cookbook workflow with schema and tests - Introduce daily paper schema types (DailyBriefOutput, PaperInfo, PaperNoteOutput, etc.) - Create daily paper cookbook module with analyze, collect, digest, rank, and select steps - Add cookbook entry point and integrate into main steps module - Replace job config export with daily brief output in schema exports - Add comprehensive unit tests covering pipeline, filtering, and output generation - Update dependencies including openai-codex and pypdf packages - Configure standalone daily paper cron job with proper scheduling and routing * test(daily_paper): update tests to use Claude Code wrapper exclusively - Add test to verify web search is disallowed by default in Claude Code - Update imports to include DailyBriefOutput, PaperNoteOutput, and PaperSelection schemas - Change test name from standalone_config_has_backend_split to reflect Claude Code only usage - Remove default agent wrapper and configure all steps to use Claude Code wrapper - Rename select_wrapper to cc_wrapper for clarity and consistency - Remove duplicate Claude Code wrapper initialization - Update test assertions to verify output schema usage matches expected sequence - Remove unused as_llm component from standalone configuration test * refactor(agent-wrapper): simplify skill resolution logic across all wrappers - Replace duplicate skill resolution code with centralized _resolve_project_skills method - Add project_path property with configurable relative path resolution - Introduce proper validation for skill names and directory existence - Change Codex wrapper to use project_path instead of workspace_path for skills - Add SKILL.md requirement validation for project skills - Remove redundant skill processing logic from individual wrappers * feat(daily_paper): add daily paper workflow with PDF analysis and brief generation - Implement shared state management and file helpers for daily-paper steps - Add PDF download and text extraction capabilities with arXiv integration - Create paper collection step with Hugging Face weekly/monthly rankings - Build ranking system using reciprocal-rank fusion with memory keyword scoring - Add Claude Code integration for paper analysis and detailed note generation - Implement digest step to create final five-minute brief from detailed notes - Add configuration for standalone daily cookbook application with cron scheduling - Create typed schema for paper information, selection, and output formats - Add atomic file writing with temporary file safety mechanisms - Implement exclusion logic for previously recommended papers and daily filters * feat(daily_paper): add DingTalk notification integration and enhance logging - Integrate DingTalk markdown send step to notify groups about daily paper briefs - Add comprehensive logging throughout daily paper workflow including start/finish events - Update daily paper analysis prompt to include code repository context requirement - Configure DingTalk notification in daily_cookbook.yaml with app credentials - Add dingtalk-stream dependency for proactive message API integration - Enhance daily paper README with DingTalk notification section and updated flow chart - Implement detailed logging for each step including paper processing and agent calls - Add test coverage for DingTalk markdown sending functionality and configuration - Update pre-commit config to exclude skills directory from checks - Add .claude/skills to gitignore for local development environment * refactor(dingtalk): move dingtalk_stream import to local scope and improve code safety - Moved global dingtalk_stream import to local scope in send.py to avoid eager loading - Added dynamic import with error handling for optional dependency cases - Updated test suite to verify lazy loading behavior works correctly - Fixed markdown title generation by using safe variable naming in wait.py - Enhanced test coverage for arxiv PDF download caching functionality - Updated application context initialization with proper resource directory configuration - Modified paper metadata to include source PDF path reference in output files * refactor(daily_paper): remove manifest system and store selection metadata in digest files - Remove JSON manifest creation and storage functionality - Store selection data directly in digest file frontmatter instead of separate manifest files - Add load_saved_selection method to rebuild selection from digest and paper-note metadata - Update README documentation to reflect new cookbook workflow architecture - Modify test cases to verify selection metadata in digest files instead of manifest JSON - Remove unused json import from multiple daily paper modules - Integrate PaperSelection schema for proper data validation in stored metadata * docs(daily_paper): add bilingual cookbook guides	2026-07-22 19:17:01 +08:00
xyf2020	630f26b119	feat(search): scoped dedup, session-chunk merge, and unified recall formatting (#384) ... * feat(search): add tool_context-scoped chunk dedup with TTL Introduce _ToolContextDedupMixin shared by search/vector_search/bm25_search to skip already-seen chunks within one agent tool_context. Per-context state lives in app_context.metadata with configurable TTL (default 24h). * feat(search): unify chunk answer rendering with merge and explicit empty messages - Refactor SearchStep/VectorSearchStep/Bm25SearchStep to share format_chunks_answer for consistent source rendering and adjacent session-chunk merging. - Distinguish empty results: ALL_RETURNED_MESSAGE when dedup removes everything vs NO_RESULTS_MESSAGE when nothing matched. - Bump JsonlFileChunker default max_chars to 4000. - Add unit tests for source-format merge and empty-result messages. * refactor(config): reorganize file_chunker components and move jsonl max_chars into config - Register explicit markdown/json/jsonl chunkers in beam.yaml and lme.yaml with markdown options (embed_toc, max_ast_sections, frontmatter handling) and jsonl max_chars=4000. - Restrict default chunker to txt/log extensions. - Revert JsonlFileChunker code default max_chars back to 2000; the 4000 value now lives in config. * chore(benchmark): increase longmemeval num_items from 64 to 500 * refactor(search): split SearchStep into simplified and v2 variants, extract counter utility - Extract global_counter_next from ApplicationContext into reme/utils/counter.py as a standalone function operating on metadata dict with lazy initialization. - Split SearchStep into two variants: - SearchStep (simplified): inline chunk.id dedup, single-branch vector/keyword optimization based on vector_weight, inline answer formatting. - SearchV2Step (full): preserves _ToolContextDedupMixin with interval-subset-aware dedup and format_chunks_answer with session-aware chunk merging. - Update beam.yaml and lme.yaml to use search_v2_step for benchmark jobs. - Rename existing search tests to test_search_v2_step_* and add new test_search_step_* tests covering the simplified variant. * fix: normalise missing trailing newline in _build_union_chunk to prevent line collision * refactor: lazy-init counter tree in ApplicationContext metadata - Remove hardcoded _counter_tree and _counter_tree_lock initialization from ApplicationContext.metadata; rely on lazy initialization in reme.utils.counter.global_counter_next on first call - Set longmemeval num_items back to 500 - Remove obsolete trailing-newline collision tests --------- Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>	2026-07-22 17:17:23 +08:00
xyf2020	7b1da5a9ee	feat(benchmark): add BEAM & restructure LongMemEval evaluation framework (#375) ... * feat(eval): add LongMemEval evaluation framework with tool_defaults date injection - Add evaluation/longmemeval/ with run.py, config.yaml, and test scripts - Add reme/config/longmemeval.yaml for evaluation-specific model config - Add tool_defaults mechanism to as_agent_wrapper for injecting default tool kwargs (uses setdefault so LLM-provided values take priority) - Pass tool_defaults={'daily_write': {'date': day}} in auto_memory to ensure notes always use the correct historical date - Add timestamp interpolation (_interpolate_timestamps) in auto_memory for filling missing created_at fields via linear interpolation - Evaluation pipeline: ingest sessions -> dream -> search -> answer -> judge - Uses qwen3.6-flash for memory, qwen3.7-max for answer/judge * chore: gitignore logs/results/demo.py, keep empty dirs * chore: update .gitignore * feat(eval): add multiprocessing and session time filtering to longmemeval runner - Replace async execution with synchronous + multiprocessing for parallel item evaluation - Add filter_future_sessions option to only ingest sessions <= question date - Add question_types filtering in config - Add result summary with binary accuracy and avg score - Update config defaults (oracle variant, 50 items, 32 workers) - Minor code style fixes in agent_wrapper and auto_memory * feat: add bench_query_step with ReAct agent for benchmark query phase - Add BenchQueryStep using agent_wrapper with search job tool - Replace manual search+LLM answer in run.py with bench_query_job - Remove unused answer LLM config from longmemeval.yaml - Register benchmark step module in steps/__init__.py * feat: add start_date/end_date time filter support for search job - Add _extract_date_from_path to extract validated YYYY-MM-DD from chunk paths - Add start_date/end_date filtering in _matches_search_filter - Implement progressive recall in FaissLocalFileStore.vector_search - Promote start_date/end_date from context to search_filter in SearchStep - Add start_date/end_date parameters to search job in default.yaml - Add unit tests for date filter functionality * fix: validate/normalize date filters and harden _extract_date_from_path Address three code-review comments on the time_filter search feature: 1. Validate/normalize start_date and end_date before string comparison. _matches_search_filter does lexicographic comparison against path_date (always canonical YYYY-MM-DD). Raw caller values like '2026-2-28' or 'abc' would produce silently wrong results. Now SearchStep normalizes valid dates via extract_daily_date (with strptime fallback for non-zero-padded input) and silently ignores invalid dates with a logger.warning, removing them from the filter. 2. Clarify behavior for paths without embedded dates. Added optional strict_date_filter parameter (default False). When True and at least one date bound is active, chunks whose path yields no date (e.g. digest/personal/topic.md) are excluded. When False (default), the existing behavior is preserved — dateless paths pass through. 3. Harden _extract_date_from_path against non-standard suffixes. Previously parts[1].split('.')[0] accepted '2026-05-18.anything' as a valid date. Now only exact 'YYYY-MM-DD' (dir) and 'YYYY-MM-DD.md' (day-index) forms are accepted. * feat(eval): LLM-as-Judge per-type prompt routing, binary-only, progress tracking - Remove 0-5 score metric, keep only binary (yes/no) classification - Load per-question-type judge prompts from llm-as-judge.json (temporal-reasoning, knowledge-update, single-session-preference, __default__) - Replace SCORE_JUDGE_PROMPT with type-specific BINARY_JUDGE_PROMPT template - judge_response(): parameter 'metric' -> 'question_type', returns single 'judgment' - Summary output: add per-type accuracy breakdown, remove score stats - Add progress tracking: background thread prints PROGRESS every 10min - Add FINAL progress line and total elapsed time on completion - Add --log-level, --reme-log-level, -q CLI arguments - Parallel mode: pool.map -> pool.imap_unordered for real-time progress - config.yaml: full oracle (10000 items), 32 workers, all question types - Add kill.sh (process cleanup) and run_async.sh (background eval launcher) * docs: add LongMemEval oracle evaluation results (61.6% accuracy) * feat(bench): add MAX_ITERATION limit to BenchQueryStep and add _auto_memory.yaml * feat: add golden session benchmark & eval_only mode with refined prompt - Add benchmark/longmemeval/run_golden_session.py for golden session evaluation - Refine PROMPTED_SYSTEM_PROMPT: concise answer rule, remove 'Information not found' fallback - Add eval_only mode to run.py (--eval_only flag) - Add multiple eval config variants (evalonly, full, test5) - Add analyze_results.py for result parsing - Update auto_memory.yaml, longmemeval.yaml, application_config - Update result-longmemeval.md with latest evaluation results - Add benchmark results to .gitignore * update: refine answer prompts and increase max iteration to 6 - Tighten prompted-answer system prompt for more concise output - Comment out 'Information not found' fallback rule - Increase MAX_ITERATION from 5 to 6 in bench_query - Add recall_eval.py - Update evaluation results * feat(chunker): add dedicated JSON and JSONL file chunkers (cherry-pick from upstream #325) - Add JsonFileChunker: structure-aware chunking preserving nested key paths, optional list-to-dict conversion, size measured by json.dumps() char count - Add JsonlFileChunker: line-aligned sliding-window chunking with configurable overlap, supports char/byte mode switching - Register both chunkers in default.yaml (json for .json, jsonl for .jsonl) - Add comprehensive unit tests (21 + 20 test cases) * feat(service): add CLI service for local job execution (from upstream #334) - Introduce CliService to execute single jobs locally without serving ports - Add prepare_start_config and should_precheck_start functions for CLI job setup - Update reme start command to use CLI service when job argument is provided - Add show_metadata to client kwargs for optional CLI metadata output - Add unit tests for CLI service functionality and configuration handling * feat(steps): add BM25/vector search steps, Python execute step, and draft steps (from upstream #334) - Add Bm25SearchStep for plain BM25 keyword search with tool_context deduplication - Add VectorSearchStep for plain vector search with tool_context deduplication - Add PythonExecuteStep to run Python code in subprocess with timeout handling - Add AddDraftStep/ReadAllDraftStep for draft accumulation scoped by tool context - Update SearchStep with tool_context dedup, dynamic default limit via REME_SEARCH_LIMIT env, and candidate_multiplier default changed from 3.0 to 5.0 - Add comprehensive unit tests for all new steps * feat(search): add tool context deduplication and improve search configuration (#321) * feat(search): add tool context deduplication and improve search configuration - Modify _make_tool methods to accept and inject tool_context_id parameter - Add tool_context_id handling in AS and CC agent wrappers - Increase search candidate multiplier from 3.0 to 5.0 in default config - Extend HTTP client timeout from 30s to 3600s - Add tool context deduplication logic to prevent duplicate search results - Implement TTL-based expiration for seen chunks in tool contexts - Add comprehensive unit tests for tool context deduplication behavior - Update .gitignore to exclude longmemeval directory - Add time import for timestamp functionality in search step * refactor(search): replace time module with datetime for timestamp generation - Removed unused time import - Added static method _now_ts using datetime.timestamp - Updated clock parameter to use _now_ts method instead of time.time - Maintained same timestamp precision and functionality * fix(file_io): fix risk of out-workspace paths (#322) * fix(file_io): fix risk of out-workspace paths * chore(file_io): remove unused unittest file * fix(as_embedding): support both agentscope 2.0.2 and 2.0.3 (#323) 2.0.3 promoted `dimensions` to a required first-class constructor argument while keeping a backfill from `parameters.dimensions`; 2.0.2 has no such argument and reads `dimensions` from `Parameters`. Keep `dimensions` in `Parameters` for both versions and, when the model constructor accepts `dimensions`, pass `dimensions=None` so 2.0.3's backfill promotes it out of `parameters`. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * Bump version to 0.4.0.7 * refactor: delegate LLM-as-Judge to answer_judge_step and update eval config/results - run.py: replace inline judge logic with judge_response_via_job using app.run_job('answer_judge') - longmemeval.yaml: expand benchmark configuration - bench_query.py: enhance benchmark query step - result-longmemeval.md: update evaluation results - judge_all_plus_results.json: add judge all-plus results * refactor: split longmemeval.yaml into lme.yaml/beam.yaml and unify job names - Split reme/config/longmemeval.yaml into lme.yaml (LongMemEval) and beam.yaml (BEAM) - Unify job names across both configs: agentic_answer, answer_judge, context_answer - Update evaluation/longmemeval/run.py and evaluation/beam/run_beam_eval.py to use unified job names - Update all evaluation config YAMLs to reference lme.yaml - Add BEAM benchmark step implementations (agentic_answer, context_answer, llm_judge) - Remove obsolete config_test5.yaml and test_5sessions.py * eval: BEAM 100K & LongMemEval cleaned-S 评测结果记录 - BEAM 100K eval-only (32并发, 20 case): Agentic 0.631, Prompted 0.468 - LongMemEval final GT (500题): Agentic 89.0%, Prompted 83.6% - 新增 benchmark/result-beam.md, benchmark/result-longmemeval.md - benchmark/beam/config.yaml: num_workers=32 * refactor: restructure benchmark directory and clean up gitignore rules - Consolidate benchmark outputs to benchmark/results/ with .gitkeep - Remove old benchmark scripts, configs and result files from benchmark/beam/ and benchmark/longmemeval/ - Add datasets/README.md and datasets/README_EN.md with download instructions - Add datasets/longmemeval/download.py and final_groundtruth_cleaned_s.json - Add memory_workspaces .gitkeep placeholders - Restructure .gitignore: fix duplicate entries, add BEAM dataset exclusion, refine logs/results ignore patterns - Remove stale result-beam.md and result-longmemeval.md from project root * chore: clean up longmemeval benchmark scripts and update dataset docs - Remove obsolete longmemeval benchmark runner/stats scripts - Update datasets/longmemeval README and add Chinese translation - Clean up final_groundtruth_cleaned_s.json * docs(benchmark): add reproduction guide for LongMemEval and BEAM - Add bilingual README for benchmark runners (EN/ZH) - Cover prerequisites, dataset download, run commands, configs, outputs, logs, and kill.sh * refactor: migrate auto_memory steps from evolve to benchmark-specific modules - Split auto_memory into beam and lme benchmark-specific implementations - Add auto_memory.py and auto_memory.yaml under steps/benchmark/beam and steps/benchmark/lme - Slim down evolve/auto_memory.py and auto_memory.yaml to shared base only - Remove obsolete evolve/_auto_memory.yaml - Update benchmark run.py, config YAMLs, and step __init__.py registrations - Update llm_judge and context_answer minor adjustments - Remove outdated test_lme_final_answer_review.py * revert(as_agent_wrapper): sync with upstream/main Remove local-only comment to keep file identical with upstream/main. * style: add trailing commas in benchmark __init__.py __all__ lists * chore: disable vector_weight range assertion in SearchStep * chore: add tests/integration/logs/ to .gitignore * refactor: replace scipy.stats.kendalltau with pure numpy implementation scipy is not listed in project dependencies. Implement Kendall's tau-b rank correlation using only numpy to remove the undeclared dependency. * feat(benchmark): add binary score metrics, update BEAM 1M results, and improve LLM retry/prompt config - benchmark/beam/run.py: add binary score calculation per rubric item and per-type/overall binary stats - benchmark/beam/config.yaml: switch to 1M dataset, reduce workers to 18 - benchmark/result-beam.md: add 1M evaluation results with binary scores - benchmark/result-longmemeval.md: minor formatting - reme/config/beam.yaml: increase max_retries to 5 and add retry_delay 5.0 for all LLM components - reme/config/lme.yaml: increase max_retries to 5 and add retry_delay for judge/prompted/bench components - reme/steps/benchmark/lme/agentic_answer.yaml: improve search strategy and answer rules prompts * fix(benchmark): fix line-too-long and add pylint disable for main() * refactor(longmemeval): use single cleaned-S dataset with embedded ground truth - Switch to agentscope-ai/ReMe_longmemeval_clean_s_v2 HuggingFace source - Remove separate final_groundtruth_cleaned_s.json (ground truth now in data file) - Simplify download.py to fetch only longmemeval_s_reme_cleaned.json - Remove dataset.variant and dataset.ground_truth_path config options - Update benchmark and datasets READMEs to reflect new workflow - Update .gitignore for new dataset filename * fix: rename loop variable to avoid pylint redefined-outer-name warning * refactor(benchmark): restructure datasets/memory_workspaces into benchmark and simplify auto_memory steps * refactor(benchmark): extract BaseAgenticAnswerStep into base module - Add reme/steps/benchmark/base/agentic_answer.py with shared agentic answer logic - Refactor beam/lme AgenticAnswerStep to inherit from BaseAgenticAnswerStep - Simplify lme/context_answer.py and update context_answer.yaml - Update result-longmemeval.md with latest evaluation results (agentic 91.0%) * refactor(benchmark): remove context_answer steps and unused configs - Remove beam/lme context_answer job definitions and step implementations - Remove prompted LLM component from beam.yaml and lme.yaml - Delete jinli_lme.yaml (no longer needed) - Simplify benchmark run.py scripts - Clean up .gitkeep files and update .gitignore - Remove unused import in search.py * chore: remove benchmark/results/.gitkeep --------- Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135> Co-authored-by: jinliyl <6469360+jinliyl@users.noreply.github.com> Co-authored-by: imrewce <wce@pku.edu.cn> Co-authored-by: Sen Huang <48879559+ployts@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-21 19:09:50 +08:00
jinliyl	e7d44f6f3b	refactor(agent): unify agent subprocess env, sessions, skills, and MCP/service jobs (#382) ... * feat(config): add environment variable configuration for agent subprocesses - Add environment field to ApplicationConfig to store variables for agent subprocesses - Remove dynamic loading of .env files in agent wrappers - Introduce subprocess_environment property in base agent wrapper - Pass application-level environment variables to Claude Code and Codex agents - Load environment variables once at startup and pass to ReMe application - Remove dependency on load_env utility in agent wrapper implementations - Update tests to use configured environment instead of dynamic loading - Remove unused environment loading utilities and related test cases * refactor(mcp): remove channel notification system and related components - Removed channel notification step implementation - Removed claim channel step implementation - Removed ChannelSink class from MCP service - Removed channel-related documentation from AGENTS.md - Removed channel instruction text from MCP service - Removed all channel-related tests - Updated application context metadata comment to remove channel sink reference - Removed channel module initialization and imports * feat(service): add job whitelisting capability to BaseService - Add optional jobs parameter to BaseService.__init__ to configure job whitelist - Store jobs as set in self.jobs attribute for efficient lookup operations - Modify add_jobs method to filter jobs based on whitelist configuration - Update documentation in both English and Chinese to describe new feature - Add comprehensive unit tests for job whitelisting behavior - Implement flowchart update showing new filtering logic - Preserve existing enable_serve flag behavior alongside new whitelisting * refactor(service): enhance service job validation and MCP tool injection - Add strict validation for service jobs whitelist with detailed error messages - Implement injected job arguments support for MCP services with conflict detection - Add tool error handling for unsuccessful responses in MCP services - Remove duplicate job names in Codex agent wrapper using dict.fromkeys - Update MCP server argument format from single JSON array to repeated --job flags - Add comprehensive test coverage for job injection and error handling scenarios - Update documentation to reflect service job validation and MCP features - Ensure application cleanup occurs even when service lifespan encounters errors * feat(agent): update skill handling to preserve existing Claude skills - Change skills parameter processing to use 'all' instead of filtered list - Add logic to select project skills without restricting Claude's existing skills - Update variable naming from 'skills' to 'selected_skills' for clarity - Modify application context metadata documentation to clarify in-memory state usage - Add test case to verify configured skills are added without filtering existing skills - Update internal skill directory handling to use renamed variable consistently * refactor(agent): restructure agent wrapper components and session storage - Move CcFileSessionStore to separate module for better organization - Add SDK package version logging in base agent wrapper - Update Claude Code agent to use new session store structure with project keys - Refactor Claude Code agent wrapper to use proper type hints and SDK integration - Add support for server tool use events in Claude Code message processing - Improve error handling and resource cleanup in streaming operations - Update Codex agent wrapper with proper type annotations and configuration - Remove deprecated system prompt mode handling from Claude Code wrapper - Fix session path construction for Claude Code transcript storage - Update dependency injection and configuration handling patterns * fix(cc_agent_wrapper): resolve Claude Code SDK integration issues - Added dataclass import and created _BlockState for content block metadata tracking - Implemented proper MCP server name constant and tool context ID validation - Fixed tool_context_id injection to prevent duplicate assignment errors - Resolved skills parameter handling in build_options method - Enhanced job tools integration with MCP servers mapping validation - Replaced deprecated block_ids/block_types/tool_call_names with block_states dict - Updated message_delta to emit USAGE chunks instead of REPLY_END - Fixed stream result handling to ensure proper REPLY_END emission - Improved error handling for session mirror failures and rate limits - Added proper cleanup for expected trailing errors in streams - Refactored Codex agent wrapper initialization and configuration management - Removed obsolete system_prompt_mode from default config - Enhanced test coverage for new block state and error handling features - Fixed async generator handling with aclosing context manager - Improved chunk type mapping for Claude Code SDK events * refactor(tests): remove demo config tests from config parser test suite - Removed test_demo_config_registers_llm_jobs function and its assertions - Eliminated verification of LLM demo job configurations - Removed checks for agent wrapper component settings - Deleted assertions for model configurations and parameters - Cleaned up deprecated test cases related to demo config parsing * refactor(evolve): simplify Claude Code session store path structure - Removed redundant project key subdirectory from session link generation - Updated CcFileSessionStore initialization to use direct session directory path - Maintained existing session layout compatibility for backward compatibility - Added unit tests to verify session persistence behavior with existing transcripts - Ensured UUID-based session files remain accessible at expected locations - Preserved existing session directory structure without additional nesting * refactor(agent): defer optional Codex SDK imports until first use - Moved openai-codex imports inside functions to avoid mandatory dependencies - Added TYPE_CHECKING guard for development time type checking only - Implemented lazy loading mechanism with _get_async_codex_class function - Updated AsyncCodex initialization to occur on demand rather than at module level - Maintained backward compatibility while improving import performance - Added test case to verify package import works without optional Codex SDK - Updated agentscope dependency to version 2.0.4.post1 in pyproject.toml * test(embedded): add compatibility tests for in-process ReMe embedding - Add test suite for QwenPaw-style embedded configurations - Verify optional defaults remain preserved in embedded configs - Ensure in-process application API stays compatible - Test model injection and lifecycle management compatibility - Remove obsolete hermes agent plugin tests - Update CLI import test to cover multiple optional SDKs - Block claude_agent_sdk and openai_codex during import testing	2026-07-20 23:52:14 +08:00
Sen Huang	55ef4bd6ad	fix(proactive): expose topics in primary answer (#380)	2026-07-20 16:05:47 +08:00
jinliyl	cf22ef3b1d	feat: add codex auth modes, background embedding/index repair, and qwenpaw logging (#371) ... * feat(codex): add authentication mode support with thread-safe logging - Implement _CodexAuthConfig dataclass for resolved auth settings - Add auth_mode parameter with auto/api_key/oauth options - Separate API key and OAuth authentication flows - Force specific login method based on auth mode - Add explicit API key validation requirement - Serialize concurrent logger initialization in thread lock - Close logging handlers properly during cleanup - Update default config with auth_mode presets for codex and codex_oauth - Add comprehensive tests for authentication modes and concurrent logging * feat(file_store): implement background embedding backfill and keyword index repair - Add _after_embedding_backfill hook in FAISS local file store - Schedule startup embedding repair without delaying component readiness - Cancel and collect embedding backfill task during component shutdown - Log progress at fixed percentage boundaries for long-running operations - Process embedding backfill in configurable batch sizes with progress reporting - Rebuild keyword index in bounded batches with detailed mismatch diagnostics - Format stdlib logs consistently with QwenPaw console output using relative paths - Run embedding backfill as background task that doesn't block component startup - Add comprehensive tests for background embedding and keyword index repair scenarios * fix(file-store): repair graph-chunk consistency on load - Add _repair_graph_chunk_consistency method to detect and fix mismatched graph/chunk states - Clear torn graph/chunk state when missing or orphaned chunks are detected - Ensure keyword index sync handles empty chunks properly - Add comprehensive tests for graph-chunk consistency scenarios - Update test utilities to properly seed graph/chunk snapshots - Increment version to 0.4.1.2 * feat(file_io): enhance list step response format and add comprehensive logging - Format list output with bullet points for better readability - Add explicit "No files found" message when directory is empty - Include detailed timing information for file store startup phases - Add logging for chunk loading, graph consistency checks, and keyword indexing - Provide detailed metrics for embedding backfill operations - Add comprehensive test coverage for empty directory scenarios - Include batch processing statistics for embedding operations * feat(logger): add QwenPaw logging integration with forwarding mechanism - Introduce _ForwardToLoggerHandler to forward log records to target logger - Add qwenpaw logger integration that forwards ReMe logs to QwenPaw handlers - Maintain ReMe logger stability for modules that cache it at import time - Enable QwenPaw handlers to take effect without ReMe reconfiguration - Add comprehensive tests for stdlib forwarding to QwenPaw sinks - Support explicit REME_DISABLE_LOGURU=false to keep original Loguru backend - Preserve existing logging behavior when QwenPaw is not configured * fix(file_store): serialize concurrent FAISS dump operations to prevent corruption - Add asyncio lock to ensure only one FAISS dump operation runs at a time - Generate unique temporary filenames using UUID tokens for atomic replacement - Implement proper cleanup of temporary files in finally block - Add comprehensive test to verify concurrent dumps are serialized - Ensure atomic writes by replacing both index and idmap files together - Prevent partial state writes during concurrent access scenarios	2026-07-20 14:47:34 +08:00
Sen Huang	1c08eaa559	fix: enforce markdown chunk byte limits (#370)	2026-07-17 22:03:58 +08:00
Sen Huang	987f275985	fix: bound markdown chunking for large section trees (#369)	2026-07-17 17:57:56 +08:00
jinliyl	c1a25e9ff4	feat(agent): add Codex agent wrapper and ReMe MCP bridge (#358) ... * feat(agent): add Codex wrapper integration * feat(agent): enhance agent wrapper functionality and add comprehensive testing - Implement structured output schema normalization across all wrappers - Add Claude Code system prompt mode support with append/replace options - Introduce Codex agent wrapper with streaming, tool context isolation, and skill management - Enhance skill linking with validation and conflict resolution - Add approval event streaming support for Codex wrapper - Implement output schema validation and normalize function - Create dedicated test suites for Claude Code and Codex integration - Update README documentation for Codex wrapper capabilities - Refactor kwargs merging with proper schema handling - Add tool context validation when resuming sessions - Implement proper cleanup and session management for Codex wrapper * test(cc-agent): add test coverage for structured output scenarios - Add docstring for empty schema validation in build_options - Document falsy structured output preservation behavior - Add docstring for streaming wrapper schema rejection - Include lambda function reference for wrapper factory consistency - Add test documentation for live Codex wrapper contract exercise * docs: revert README changes * fix(agent): interrupt abandoned Codex turns	2026-07-17 13:39:18 +08:00
jinliyl	329fd9a6a6	refactor(config): remove max_file_bytes limit from background jobs (#367) ... - Removed max_file_bytes configuration from index_update_loop, resource_watch_loop, digest_watch_loop, and reindex jobs - Updated default.yaml to reflect simplified job configurations without file size limits - Removed corresponding test case that validated the 20 MiB limit behavior - Simplified watch directories and suffixes to basic configurations - Cleaned up unnecessary commented configurations in the YAML file	2026-07-17 11:26:39 +08:00
jinliyl	2eb05392c6	chore(benchmark): remove longmemeval final answer review file (#366) ... * feat(benchmark): add final answer review step for evaluation - Introduce FinalAnswerReviewStep to handle answer validation - Add final_answer_review.jsonl dataset with 24 evaluation cases - Include detailed reasoning and golden check results for each case - Support various question types including temporal reasoning and preferences - Implement time consistency checks for session references - Add comprehensive test coverage for different evaluation scenarios * chore(benchmark): remove longmemeval final answer review file - Removed final_answer_review.jsonl containing 23 evaluation records - Deleted question_id mappings with detailed reasoning for golden answers - Removed answer correctness assessments and session time validation checks - Cleaned up benchmark dataset used for memory evaluation testing - Eliminated JSONL format evaluation results for temporal reasoning tasks - Removed references to various session IDs and time-based validations * config(default): disable shell step configuration by commenting out - Commented out the shell step configuration in default.yaml - Disabled asynchronous shell command execution capability - Removed shell step from available backend operations - Preserved traverse backend configuration unchanged * refactor(tests): remove unused shell job test from config parser tests - Removed test_default_config_registers_shell_job function that was no longer needed - Kept existing test for frontmatter chunk metadata configuration - Cleaned up test suite by removing obsolete test case	2026-07-16 20:32:28 +08:00
jinliyl	c3b1e93918	feat(index): add file size limits and oversized file handling (#362) ... * feat(index): add file size limits and oversized file handling - Implement max_file_bytes configuration option for content processing jobs - Add default 20MB file size limit for background processing in default config - Skip oversized files during auto_resource step with appropriate metadata - Clear stale index entries when oversized files are modified - Add size-based filtering logic to update_changes step with skip reporting - Include file size validation in UpdateIndexStep with proper response handling - Add comprehensive tests for oversized file scenarios in auto_resource and update_index - Document file size limits in constants with appropriate thresholds * chore(version): bump version to 0.4.1.1 - Update __version__ from 0.4.1.0 to 0.4.1.1 in __init__.py * fix(index): isolate batch metadata and handle file races	2026-07-15 21:01:18 +08:00
jinliyl	2e87b7a52e	feat(core): add shell execution and runtime memory status (#344) ... * feat(core): add shell command execution and memory status reporting - Introduce ShellStep for executing shell commands with timeout support - Add StatusStep to report memory estimates for stateful data components - Register shell and status commands in default configuration - Update documentation with new reme status and shell command capabilities - Implement comprehensive unit tests for both new step types - Add support for asynchronous command execution with proper error handling * feat(config): add log_config option to suppress config loading logs - Add log_config parameter to resolve_app_config function with default True - Conditionally log config loading messages based on log_config flag - Update reme.py and service_utils.py to use log_config=False for client calls - Suppress config logging in user-facing contexts to avoid output pollution refactor(shell): rename command parameter to cmd for clarity - Change 'command' to 'cmd' in default.yaml configuration schema - Rename 'timeout' to 'shell_timeout' to avoid parameter name collisions - Update ShellStep to accept both legacy and new parameter names - Maintain backward compatibility with existing command/timeout usage test(shell): add comprehensive tests for shell step parameter handling - Add test cases for new cmd and shell_timeout parameter names - Verify legacy command and timeout parameters still work - Test blank command rejection message updated to use cmd - Create integration test for shell parameter payload passing * fix(shell): ensure proper environment loading and process timeout handling - Move load_env() call to execute before parse_args() in main function - Add proper process group killing for timeout scenarios on POSIX systems - Implement recursive child process termination on Windows for proper cleanup - Change parameter name from 'timeout' to 'shell_timeout' in shell execution - Remove support for legacy 'command' and 'timeout' parameter names - Update test cases to verify new timeout behavior and parameter requirements - Add comments explaining component size tracking implementation details	2026-07-14 16:31:41 +08:00
jinliyl	bf7ca17705	feat(benchmark): add LongMemEval golden answer validation (#335) ... * feat(benchmark): add golden answer validation and session review for LongMemEval - Introduce GoldenCheckStep to validate LongMemEval golden answers using structured verdicts - Add SessionReviewStep to extract query/answer-relevant evidence from all sessions - Implement concurrent session processing with configurable concurrency limits - Create check_golden job configuration with lme_review and lme_judge agent wrappers - Add Qwen3.7-plus model configuration for enhanced processing capabilities - Include python_execute tool integration for agent-based reasoning and date validation - Generate comprehensive JSON output with session summaries and validation verdicts - Add run_check_golden.py script for batch processing across all LongMemEval samples - Configure proper logging initialization with console and file output options - Update component registry and file I/O modules to support new benchmark features * feat(scripts): add script to summarize LongMemEval check_golden verdicts - Parse check_golden.json files across all LongMemEval samples - Calculate accuracy metrics for golden answers and session IDs - Provide breakdown by question type with percentage calculations - Add command line options for listing bad samples and JSON output - Include progress tracking showing completed vs pending samples - Display confidence scores and date sanity checks statistics * refactor(benchmark): move golden check scripts to longmemeval directory - Moved run_check_golden.py from scripts/ to benchmark/longmemeval/ - Moved stats_check_golden.py from scripts/ to benchmark/longmemeval/ - Updated path resolution to use parents[2] instead of parent.parent - Added new --list-run-failed option to stats script - Added logging directory constant and functions for tracking launched samples - Enhanced stats output with launched count and run failure information - Improved error reporting with run failure details and log file paths * feat(benchmark): add LongMemEval agentic answer workflow with session extraction - Add LmeAgenticAnswerStep, LmeAutoMemoryStep, and LmeExtractSessionStep to __init__.py - Create shared helper render_with_source for displaying search results with session_id - Implement agentic_answer step with vector_search, bm25_search, and extract_session_by_id tools - Add auto_memory step to convert each session into search-friendly daily notes - Create extract_session step to retrieve and analyze raw session content by session_id - Update jinli_lme.yaml with auto_memory, vector_search, bm25_search, and agentic_answer jobs - Configure lme_memory, lme_extract, and lme_agentic_answer agent wrappers - Enhance search steps with include_source option to show session_id metadata - Add proper session_id tracking and collision handling in daily note generation * feat(benchmark): add LongMemEval agentic answer evaluation pipeline - Add session_id tracking to agentic_answer.py result metadata - Introduce run_agentic_answer.py driver for complete pipeline execution - Implement auto_memory, update_index, and agentic_answer job orchestration - Add concurrent execution with configurable limits and staggering - Create aggregation script for collecting tool-call trails and results - Add stats_agentic_answer.py for comprehensive result analysis - Implement resume capability with existing output detection - Generate aggregate.json with per-sample breakdown and tool call summaries * feat(steps): add ClearPathsStep for cleaning workspace outputs before rebuild - Introduce ClearPathsStep to remove stale workspace files/directories - Add support for specifying paths and config_keys as targets to clear - Implement safety checks to prevent deletion of files outside workspace - Add logging for cleared paths and warnings for invalid paths - Configure clear_paths_step in jinli_lme.yaml to clean daily_dir - Add clear_paths_step to clean mem_answer.json before rebuilds * feat(benchmark): add resume functionality to agentic answer runner - Replace --force flag with --resume flag for controlling job execution - By default every job reruns with clean rebuild behavior using config clear steps - Add --resume option to skip samples whose output already exists and continue interrupted batches - Update documentation to reflect new default clean rebuild behavior - Modify job skipping logic to honor resume flag instead of force flag - Update dry-run output to show correct todo jobs based on resume status - Change default example command to use --resume for continuing interrupted runs * feat(benchmark): generate JSONL output for check golden records - Add write_check_golden_list function to create JSONL file - Write all readable check_golden records as JSONL format - Include check_golden_list path in stats output - Display generated JSONL file path in summary report - Maintain UTF-8 encoding with non-ASCII character support * refactor(benchmark): rename answer judge step and integrate LME LLM judge - Rename AnswerJudgeStep to LmeLlmJudgeStep and update imports - Add new llm_judge configuration in jinli_lme.yaml - Update run_agentic_answer.py to include llm_judge in pipeline - Modify LmeLlmJudgeStep to read from query.json and answer.json - Write LLM judgement results back to mem_answer.json - Add command line options for start/end sample range selection - Update aggregate.json generation to include LLM judgement data - Add resume capability for llm_judge job based on judgement presence * refactor(benchmark): rename answer judge step and integrate LME LLM judge - Rename AnswerJudgeStep to LmeLlmJudgeStep and update imports - Add new llm_judge configuration in jinli_lme.yaml - Update run_agentic_answer.py to include llm_judge in pipeline - Modify LmeLlmJudgeStep to read from query.json and answer.json - Write LLM judgement results back to mem_answer.json - Add command line options for start/end sample range selection - Update aggregate.json generation to include LLM judgement data - Add resume capability for llm_judge job based on judgement presence * feat(steps): add wait_for_paths_step to block until workspace files exist - Introduce WaitForPathsStep class that polls for required workspace-relative paths - Add step registration with 'wait_for_paths_step' backend identifier - Implement path validation to ensure targets are within workspace boundaries - Add polling mechanism with configurable intervals via poll_seconds parameter - Include logging functionality with log_every_seconds parameter for status updates - Add metadata tracking of waited paths and duration in response object - Register step in index module and expose in public API - Configure step in jinli_lme.yaml to wait for session_review.json before golden check - Add script rename from run_check_golden.py to run_golden_check.py with enhanced options * feat(benchmark): enhance longmemeval benchmarking with concurrency and progress tracking - Add benchmark extra dependency group with portalocker requirement - Introduce concurrent execution support for golden_check and session_review workflows - Add progress reporting interval option with real-time status updates - Implement global throttling mechanism for session review requests using file locks - Enhance golden check validation with current schema verification - Add active task tracking and graceful shutdown handling - Rename check_golden scripts to golden_check for consistency - Update statistics reporting with correct/incorrect terminology instead of reasonable - Add stale format detection and compatibility handling for verdict fields - Include both_correct rate calculation in accuracy metrics - Add concurrency and staggering options for better resource management * ci(workflow): add Windows smoke test workflow - Create new workflow file .github/workflows/windows-smoke.yml - Configure workflow to trigger on push and pull request events - Set up Python environment with version 3.11 - Install package dependencies using pip - Run version job as smoke test for CLI functionality - Enable concurrency control to prevent duplicate runs - Use matrix strategy for Python version testing * feat(benchmark): add retry mechanism and health check for session review - Added retry configuration options (retry_initial_seconds, retry_max_seconds, retry_max_attempts) to jinli_lme.yaml - Implemented exponential backoff retry logic with configurable parameters in session_review step - Added output_is_healthy function to verify session_review.json integrity and absence of failed reviews - Updated resume functionality to skip only healthy outputs instead of all existing files - Integrated JSON parsing and validation to check for failed reviews in output files - Enhanced error handling and logging for retry attempts and recovery scenarios * feat(benchmark): add LongMemEval session review statistics script - Create stats_session_review.py to summarize session_review.json artifacts - Add command line options for listing failed, missing, and run failed samples - Implement JSON output mode for programmatic consumption - Calculate and display health statistics including total samples, healthy outputs, failed sessions - Provide detailed failure information with session IDs and error messages - Generate re-run commands for samples with failed reviews - Add percentage calculations for better statistical overview - Include support for multiple output formats and detailed logging * feat(benchmark): add LongMemEval output cleanup script and enhance golden check retry logic - Added clean_sample_outputs.py script to remove generated LongMemEval files while preserving source inputs - Implemented configurable retry mechanism in golden_check.py with exponential backoff strategy - Added retry parameters (initial/max seconds and max attempts) to control failure recovery behavior - Integrated asyncio support for asynchronous sleep during retry intervals - Configured default retry settings in jinli_lme.yaml with 5s initial and 300s maximum intervals - Preserved core files (query.json, answer.json, session/) while cleaning generated artifacts * feat(benchmark): add AppleDouble file cleanup to sample output cleaner - Remove AppleDouble files starting with '._' recursively including under session/ - Add is_under helper function to check if path is inside parent directory - Track targets in set to avoid duplicate processing - Include AppleDouble files in cleanup targets when not already covered by existing targets - Maintain dry-run mode as default behavior with --apply flag for actual deletion * refactor(benchmark): update LongMemEval sample output cleaning script - Add time and Iterator imports for enhanced functionality - Add --progress-every argument to control progress reporting frequency - Replace is_under function with iter_sample_targets generator - Implement detailed progress tracking with timing measurements - Add sample-by-sample processing with elapsed time reporting - Include AppleDouble file detection within session directory - Update target counting and deletion statistics display - Add conditional progress updates based on progress-every setting - Improve dry-run mode with would-delete indication * chore(benchmark): increase initial interval for session review step - Changed START_INTERVAL_SECONDS from 1.0 to 3.0 seconds - Adjusted timing parameters for better benchmark stability * refactor(benchmark): implement coordinated retry mechanism for session reviews - Add retry gate condition to coordinate concurrent review attempts - Implement wait_for_healthy_start_slot to handle sequential retries - Create mark_retrying and mark_recovered functions to track retry states - Update reply_with_retry to accept index parameter for coordination - Add has_prior_retry logic to prevent race conditions during recovery - Ensure proper cleanup of retry state on success or failure - Maintain backward compatibility while adding coordination features * chore(benchmark): adjust session review start interval timeout - Changed START_INTERVAL_SECONDS from 3.0 to 5.0 seconds - Increased initial delay for session review benchmark step - Updated timeout configuration for improved stability * refactor(benchmark): update session review concurrency and throttling mechanism - Replace global throttle with per-process concurrency control - Add concurrency parameter with default value of 30 in config - Add start_interval_seconds parameter with default value of 2 seconds - Change default concurrency from 3 to 1 in command line interface - Update documentation to reflect new throttling behavior - Implement semaphore-based concurrency limiting for review tasks - Modify retry mechanism to use local locking instead of global files - Remove portalocker dependency for cross-process throttling * refactor(config): update session review configuration and concurrency settings - Removed deprecated retry configuration parameters from jinli_lme.yaml - Increased MAX_CONCURRENCY from 30 to 60 in session_review.py - Reduced START_INTERVAL_SECONDS from 2.0 to 1.0 in session_review.py - Cleaned up redundant backend specifications in configuration file - Simplified agent wrapper configurations by removing obsolete retry settings * feat(benchmark): enhance LME auto memory step with advanced scheduling and error handling - Add datetime parsing functionality for LongMemEval timestamps with regex pattern - Implement configurable concurrency limits with MAX_CONCURRENCY of 60 - Introduce retry mechanism with exponential backoff for agent interactions - Add session filtering based on date comparison with question_date validation - Create rate limiting with start interval control between requests - Implement sophisticated retry coordination using asyncio conditions - Add comprehensive error tracking for failed and filtered session extracts - Remove deprecated concurrency parameter from jinli_lme.yaml configuration - Add structured output validation in session review step - Include detailed metadata reporting with session statistics and errors * fix(benchmark): adjust default concurrency for auto_memory job - Changed default concurrency from 3 to 1 for auto_memory job to prevent API overload - Updated help text to reflect new default value of 1 for concurrency parameter - Modified documentation to clarify concurrency behavior varies by job type * refactor(search): replace hardcoded candidate multiplier with constant - Introduced _CANDIDATE_MULTIPLIER constant set to 10 - Replaced hardcoded factor of 5 with _CANDIDATE_MULTIPLIER in BM25 search - Replaced hardcoded factor of 5 with _CANDIDATE_MULTIPLIER in vector search - Updated test to verify both search steps use ten times limit for candidates - Imported VectorSearchStep and Bm25SearchStep in test module - Added comprehensive test case for candidate count calculation logic * feat(lme): add data inspection error handling with fallback mechanism - Implemented non-retryable data inspection error markers detection - Added _is_data_inspection_error method to identify inspection failures - Created fallback handling for data inspection errors in auto memory extraction - Added fallback handling for data inspection errors in session review - Extended failed extracts tracking with non-retryable and fallback flags - Separated fallback extracts from regular failed extracts in reporting - Enhanced error logging with specific data inspection failure messages - Updated metrics to track fallback extractions and reviews separately - Maintained existing retry logic for other exception types * feat(benchmark): enhance session review statistics with fallback tracking - Add support for identifying and listing non-retryable fallback reviews - Introduce --list-fallback argument to display fallback review details - Separate retryable failures from non-retryable fallbacks in reporting - Track fallback samples and sessions separately from failed ones - Update console output to show both retryable and non-retryable categories - Include fallback details in JSON output with reasons and session info - Modify failure counting logic to distinguish between retryable and fallback reviews * feat(benchmark): add question_id tracking and enhanced fallback reporting - Add question_id function to extract query.question_id from data - Initialize question_id_by_id dictionary to store question IDs by index - Store question_id for each sample during data processing - Enhance fallback output to include question IDs and session information - Format sample labels with question IDs when available - Display session IDs associated with each fallback case * feat(benchmark): add question_id support and improve bad sample reporting - Add question_id_for function to extract question_id from multiple sources - Add sample_label function to format samples as idx(question_id) when available - Store question_id in data dictionary during processing - Change bad_golden and bad_sessions to store full records instead of just indices - Update list_bad output to show formatted labels with question_id information - Improve error reporting with more detailed sample identification * feat(benchmark): enhance golden check stats with structured output - Add related_session_ids function to extract session IDs from verdict records - Create grouped_records function to group records by question type - Replace flat list output with JSON-formatted grouped records in list_bad option - Replace flat list output with JSON-formatted grouped records in list_bad_sessions option - Maintain Chinese labels while adding structured data presentation - Improve readability of bad verdict record display with hierarchical grouping * feat(benchmark): update data structure for question indexing - Replace sample_label with _idx field for index tracking - Add question_id field to store _question_id values - Maintain backward compatibility with empty string defaults - Preserve existing session_id functionality - Update data mapping to include new fields in grouped results * refactor(benchmark): streamline golden answer verification process - Replace relevance filtering with comprehensive information extraction - Remove is_relevant field and simplify session summary structure - Change relevant_info to extracted_info for clarity - Update golden check logic to work with full extractions instead of filtered summaries - Simplify prompt instructions to focus on complete information extraction - Remove redundant schema validation and structured output requirements - Adjust statistics calculation to match new extraction approach - Update metadata field names to reflect extraction rather than relevance checking * feat(benchmark): add selective file deletion option to clean_sample_outputs - Add --filename argument to delete only specific root-level files - Modify iter_sample_targets function to accept optional filenames filter - Implement validation for root-level filename constraints - Update function calls to pass filenames parameter - Add example usage for selective file deletion in documentation * feat(benchmark): add error count metrics to golden check statistics - Added golden_bad, session_bad, and both_bad calculation fields - Updated console output format to include error counts per question type - Modified table display to show both accuracy rates and error numbers - Enhanced statistical summary with additional error breakdown metrics * test(search): update search step tests with include_source parameter - Added include_source=False parameter to VectorSearchStep initialization - Added include_source=False parameter to Bm25SearchStep initialization - Maintained existing RuntimeContext parameters for both search steps - Updated test calls to match new constructor signature with include_source option	2026-07-13 21:26:27 +08:00
jinliyl	90e7adc2d2	chore(config): disable embeddings by default and update documentation (#341) ... - Set default version to 0.4.1.0 - Comment out embedding configuration in default.yaml - Update README and README_ZH to clarify embedding components are disabled by default - Add note explaining how to enable embedding-based semantic retrieval - Adjust table formatting and descriptions in documentation - Modify search command description to reflect vector search availability when enabled	2026-07-13 21:57:54 +09:00
Ziyang Guo	c5eefe4da3	fix(search): expose markdown frontmatter on chunks (#314) ... * fix(search): expose markdown frontmatter on chunks * style(search): apply pre-commit formatting * fix(search): make frontmatter chunk metadata opt-in * fixup! fix(search): expose markdown frontmatter on chunks * feat(markdown): add include_frontmatter_keys_in_metadata allow-list opt-in --------- Co-authored-by: RerankerGuo <1875366113@qq.com> Co-authored-by: Ziyang Guo <121015044+RunMarshal@users.noreply.github.com>	2026-07-08 17:59:55 +09:00
jinliyl	2612d25959	feat(lme): add cli execution and agentic search tooling (#334) ... * feat(service): add CLI service for local job execution - Introduce CliService to execute single jobs locally without serving ports - Add prepare_start_config and should_precheck_start functions for CLI job setup - Update reme start command to use CLI service when job argument is provided - Change default service backend from http to cli in jinli_lme config - Modify SearchStep to use constants and rename configuration parameters - Add unit tests for CLI service functionality and configuration handling - Update file extension support to include json format in addition to md and jsonl * feat(search): add BM25 and vector search steps with configuration updates - Add Bm25SearchStep and VectorSearchStep classes with tool context deduplication - Register new search step components in index module - Update configuration to use separate vector_search and bm25_search endpoints - Modify LLM models from qwen3.7-plus/glm-5.1 to glm-5.2 variants - Adjust search parameters and remove hybrid search implementation - Configure embedding store as default in storage settings - Remove auto-memory and file catalog configurations - Update watch directories from multiple paths to session_dir only * feat(agent): add tool result offloading and workspace management - Add tool_results_dir configuration option for offloaded tool results storage - Implement ToolResultOffloadMiddleware to persist large tool results to files - Create WorkspaceBackend to standardize file operations across tools - Add configurable builtin tools selection with sequential execution option - Integrate middleware support for agent wrapper with offloading capability - Update application initialization to create tool results directory - Add safety mechanisms for filesystem operations with sanitized filenames - Enhance agent wrapper with configurable working directory handling - Upgrade agentscope dependency to version 2.0.4 for improved features # Conflicts: # reme/application.py * feat(benchmark): add LongMemEval agentic search and result management - Introduce AgenticAnswerStep for agent-based history search - Add LmePrepareJudgeStep and LmeSaveResultStep for evaluation pipeline - Implement AddDraftStep and ReadAllDraftStep for evidence accumulation - Update configuration with new agent wrapper and search parameters - Add comparison script for analyzing agent run differences - Include documentation for LongMemEval failure analysis - Enhance tool result offloading with skip options - Modify search defaults and indexing behavior * feat(agent): implement tool result offloading with system reminders - Added tool_result_offload_message parameter to agent wrapper reply method - Implemented configurable reminder template for offloaded tool results - Created system reminder messages when tool results are offloaded to files - Added Chinese user message template for agentic answer step - Updated tool result offloading middleware to use custom reminder templates - Enhanced agentic answer instructions to handle long tool results via draft storage * feat(scripts): add LongMemEval results summarization tool - Create summarize_lme_results.py script to analyze result JSON files - Implement command line interface with answer id and dataset root options - Add support for specifying index range with start and end parameters - Include option to show failure details and non-successful completions - Calculate completion statistics and accuracy metrics - Display detailed breakdown of yes/no/other judgements - Handle missing and unreadable result files gracefully - Format output with percentages and comprehensive summary statistics * feat(summarize_lme_results): add question type breakdown to result summary - Import defaultdict from collections module - Add by_type dictionary to track statistics by question type - Count completed, yes, no, and other responses for each question type - Display detailed breakdown table showing accuracy by question type - Include question type column when processing judgements - Print comprehensive summary with question type distribution - Calculate and display accuracy percentage for each question type category * feat(lme): switch to qwen3.7-max model and add shuffle functionality - Changed default LLM model from glm-5.1 to qwen3.7-max in jinli_lme.yaml - Added random module import for shuffle functionality - Implemented --shuffle argument with BooleanOptionalAction for dataset shuffling - Added --seed argument to control random seed for reproducible shuffling - Applied random shuffle to dataset indices when shuffle is enabled - Added console output showing shuffle operation and seed information * fix(cli): set default random seed for shuffle functionality - Changed default seed value from None to 42 for consistent shuffling behavior - Ensures reproducible results when using shuffle option without explicit seed - Maintains backward compatibility while providing deterministic defaults * refactor(benchmark): update agentic answer guidelines for grounding - Updated English instruction to emphasize strict grounding in retrieved context - Modified Chinese instruction to stress evidence-based responses without inference - Removed redundant conciseness requirement in both language versions - Enhanced clarity on proper use of draft saving and retrieval mechanisms - Strengthened emphasis against hallucination of unsupported facts * refactor(benchmark): update agentic search instructions and configuration - Replace separate vector_search and bm25_search with unified search tool - Update agent instructions to use single search tool with multiple strategies - Simplify Chinese instructions for search methodology - Add comprehensive search tool configuration with hybrid vector/BM25 capabilities - Increase model retry attempts from 1 to 3 for better reliability - Remove redundant tool references from job_tools list * feat(search): add configurable search limit with environment variable support - Remove hardcoded limit and min_score parameters from config schema - Increase LLM context size from 200000 to 1000000 - Add REME_SEARCH_LIMIT environment variable support for search configuration - Implement command line argument --search-limit to override default search limit - Add input validation to ensure search limit is positive - Modify subprocess execution to pass environment variables - Update search step to use dynamic default limit from environment or fallback to 5 * refactor(benchmark): remove agentic answer step and related configurations - Removed AgenticAnswerStep class and its registration - Deleted agentic_answer.yaml prompt configuration file - Removed agentic answer related job definitions from jinli_lme.yaml - Cleaned up tool result offloading middleware implementation - Removed tool_results_dir configuration field from application config - Deleted comparison and analysis scripts for agent runs - Removed agentic answer step from LME init module exports - Updated agent wrapper to remove tool result offloading functionality - Removed unused imports and dependencies in agent wrapper module * refactor(benchmark): remove unused LME result processing components - Removed LmePrepareJudgeStep and LmeSaveResultStep classes from benchmark module - Cleaned up imports and exports in lme module initialization - Removed unused middleware configuration from agent wrapper - Deleted obsolete result.py file containing deprecated result processing logic - Simplified agent instantiation by removing middleware parameter - Updated import statements to reflect removed dependencies * refactor(index): remove unused search steps and update imports - Remove Bm25SearchStep and VectorSearchStep from index steps module - Remove unused prepare_start_config and should_precheck_start exports - Move import statements to proper location in reme.py - Update test module to use direct import path for CliService - Remove vector_search and bm25_search configurations from jinli_lme.yaml - Add workspace directory environment variable configuration - Add docstring to getcwd method in agent wrapper - Remove empty middleware list from agent wrapper initialization * feat(index): add BM25 and vector search steps with tool context deduplication - Add Bm25SearchStep for plain BM25 keyword search with tool_context deduplication - Add VectorSearchStep for plain vector search with tool_context deduplication - Implement tool context state management with TTL-based deduplication - Add support for chunk deduplication across tool contexts within TTL window - Update index steps module to include new search step classes - Add test coverage for CLI metadata output functionality - Refactor CLI service to remove unused show_status parameter - Update documentation comments to reflect internal service configuration * feat(steps): add Python code execution capability - Introduce PythonExecuteStep to run Python code in subprocess - Add configuration for python_execute step in jinli_lme.yaml - Register python_execute in available tools list - Implement timeout handling with default 60 second limit - Capture stdout/stderr output and return code metadata - Add comprehensive unit tests for execution scenarios - Support workspace directory context for code execution - Handle timeout errors and runtime exceptions gracefully * refactor(python_execute): replace subprocess with asyncio for Python code execution - Replace subprocess.run with asyncio.create_subprocess_exec for non-blocking execution - Add _PythonResult dataclass to encapsulate execution results and timeout status - Implement proper timeout handling with asyncio.wait_for and process.kill() - Update metadata to include returncode and stderr when timeout occurs - Convert synchronous _run_python method to asynchronous implementation - Maintain backward compatibility while improving execution reliability * refactor(python_execute): replace subprocess with asyncio for Python code execution - Replace subprocess.run with asyncio.create_subprocess_exec for non-blocking execution - Add _PythonResult dataclass to encapsulate execution results and timeout status - Implement proper timeout handling with asyncio.wait_for and process.kill() - Update metadata to include returncode and stderr when timeout occurs - Convert synchronous _run_python method to asynchronous implementation - Maintain backward compatibility while improving execution reliability	2026-07-08 17:43:09 +09:00
xyf2020	82971ac5b0	feat(chunker): better json chunker and json chunker (#325) ... * feat(file_chunker): add dedicated JSON and JSONL file chunkers - Add JsonFileChunker: structure-aware chunking preserving nested key paths, optional list-to-dict conversion, size measured by json.dumps() char count - Add JsonlFileChunker: line-aligned sliding-window chunking with configurable overlap, supports char/byte mode switching - Register both chunkers in default.yaml (json for .json, jsonl for .jsonl) - Add comprehensive unit tests (21 + 20 test cases) * chore(config): update default chunker supported_extensions to txt/log * refactor(json_chunker): optimize _build_tree O(n²) serialization and rewrite tests - Fix O(n²) redundant json.dumps in _build_tree: * Empty containers handled directly as leaves (0 serialization) * Non-empty containers recurse first, then reconstruct+dump once * Only containers that become leaves pay serialization cost - Add _reconstruct_object/_reconstruct_array helpers - Remove dead code: _merge_json method - Apply user changes: min_element_size formula 0.01->0.05, threshold < to <= - Use indent=None for compact output (consistent with _SizeNode estimation) - Remove unused _text_size from JsonlFileChunker Test rewrite: - Replace try/finally boilerplate with make_json fixture - Group tests into TestXxx classes with pytest.mark.parametrize - Add TestOutputValidation: 9 parametrized scenarios verifying: * All chunks are valid JSON * Text length <= chunk_chars (with single-leaf tolerance) * Leaf-value concatenation matches original data (dict + array roots) - Add TestSizeNode: incremental size accuracy tests - Add TestDfsAlgorithm: path wrapping, DFS order, calibration tests - Update test_min_element_size_formula for new 0.05 multiplier - Update test_build_tree_structure for larger min_element_size * chore: apply black formatting to test files	2026-07-08 15:18:59 +08:00
jinliyl	38cf16071b	refactor(embedding): update embedding model initialization and session storage paths (#329) ... * refactor(embedding): update embedding model initialization and session storage paths - Remove unused inspect import from as_embedding module - Pass dimensions directly to embedding model constructor instead of using parameters - Update session state file paths to use mem_session directory instead of resource - Add mem_session_dir configuration option to application config schema - Update workspace directory creation to include new mem_session directory - Change AgentScope and Claude Code session paths to use mem_session directory - Move embedding dimensions from parameters to top-level configuration - Update AgentScope dependency version from 2.0.3 to 2.0.4 - Update integration tests to reflect new session file location paths * chore(version): bump version to 0.4.0.8 - Update __version__ from 0.4.0.7 to 0.4.0.8 in __init__.py	2026-07-08 12:30:46 +09:00
jinliyl	10da205797	feat(benchmark): add lme benchmark steps (#326) ... * refactor(search): replace time module with datetime for timestamp generation - Removed unused time import - Added static method _now_ts using datetime.timestamp - Updated clock parameter to use _now_ts method instead of time.time - Maintained same timestamp precision and functionality * test(http): add tests for HTTP client display formatting - Add test for default metadata hiding behavior in CLI output - Add test for metadata display when show_metadata is enabled - Verify _format_for_display method correctly formats response text - Test both success case and metadata inclusion scenarios * chore(build): remove longmemeval from gitignore - Removed longmemeval directory from gitignore list - Kept evaluation and datasets directories in ignore list - Updated gitignore configuration for proper version control	2026-07-07 18:53:39 +09:00
jinliyl	7369342115	feat(search): add tool context deduplication and improve search configuration (#321) ... * feat(search): add tool context deduplication and improve search configuration - Modify _make_tool methods to accept and inject tool_context_id parameter - Add tool_context_id handling in AS and CC agent wrappers - Increase search candidate multiplier from 3.0 to 5.0 in default config - Extend HTTP client timeout from 30s to 3600s - Add tool context deduplication logic to prevent duplicate search results - Implement TTL-based expiration for seen chunks in tool contexts - Add comprehensive unit tests for tool context deduplication behavior - Update .gitignore to exclude longmemeval directory - Add time import for timestamp functionality in search step * refactor(search): replace time module with datetime for timestamp generation - Removed unused time import - Added static method _now_ts using datetime.timestamp - Updated clock parameter to use _now_ts method instead of time.time - Maintained same timestamp precision and functionality	2026-07-06 16:18:39 +09:00
xyf2020	43a407bc4f	feat: add start_date/end_date time filter support for search job (#317) ... * feat: add start_date/end_date time filter support for search job - Add _extract_date_from_path to extract validated YYYY-MM-DD from chunk paths - Add start_date/end_date filtering in _matches_search_filter - Implement progressive recall in FaissLocalFileStore.vector_search - Promote start_date/end_date from context to search_filter in SearchStep - Add start_date/end_date parameters to search job in default.yaml - Add unit tests for date filter functionality * fix: validate/normalize date filters and harden _extract_date_from_path Address three code-review comments on the time_filter search feature: 1. Validate/normalize start_date and end_date before string comparison. _matches_search_filter does lexicographic comparison against path_date (always canonical YYYY-MM-DD). Raw caller values like '2026-2-28' or 'abc' would produce silently wrong results. Now SearchStep normalizes valid dates via extract_daily_date (with strptime fallback for non-zero-padded input) and silently ignores invalid dates with a logger.warning, removing them from the filter. 2. Clarify behavior for paths without embedded dates. Added optional strict_date_filter parameter (default False). When True and at least one date bound is active, chunks whose path yields no date (e.g. digest/personal/topic.md) are excluded. When False (default), the existing behavior is preserved — dateless paths pass through. 3. Harden _extract_date_from_path against non-standard suffixes. Previously parts[1].split('.')[0] accepted '2026-05-18.anything' as a valid date. Now only exact 'YYYY-MM-DD' (dir) and 'YYYY-MM-DD.md' (day-index) forms are accepted.	2026-07-03 15:58:04 +08:00
Ziyang Guo	c060933e4d	fix(auto-memory): preserve message timestamps (#310) ... * fix(auto-memory): preserve message timestamps * fix(auto-memory): infer daily date from messages * feat(file_io): add strict date parsing and improve daily date handling - Add new parse_daily_date function for strict YYYY-MM-DD validation - Replace extract_daily_date with parse_daily_date for explicit date validation - Change _messages_day to use max date instead of min for historical imports - Reorder imports to maintain consistent module ordering - Move session message saving after date validation in auto_memory - Add comprehensive tests for invalid date rejection before saving - Add tests for strict YYYY-MM-DD date format validation - Update test names to reflect latest date behavior --------- Co-authored-by: Ziyang Guo <121015044+RunMarshal@users.noreply.github.com> Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>	2026-07-01 17:15:07 +08:00
Sen Huang	435aa713a2	fix(config): correct indentation in default.yaml (#308)	2026-07-01 12:13:21 +08:00
Sen Huang	3dee10d4f9	feat: add Claude Code plugin with auto-memory functionality (#297) ... * feat: add Claude Code plugin with auto-memory functionality * refactor(auto_memory): fix spacing in json parsing logic	2026-06-26 14:46:42 +08:00
jinliyl	ffb4d08c4f	feat(mem): Enhance daily note system with metadata handling and write functionality (#295) ... * feat(file_io): add daily_write step for creating daily notes with conversation metadata - Add DailyWriteStep class that delegates to write job for creating daily notes - Register daily_write job in default configuration with proper parameters - Include validation for name and session_id path components - Add test coverage for daily_write functionality including metadata handling - Preserve existing job execution method in application.py after repositioning - Update base_step.py to use positional-only parameter syntax for job methods - Import and expose DailyWriteStep in file_io module initialization - Override reserved metadata keys (name, description, session_id, source_conversation) with fixed values - Refresh daily index after successful write operation - Generate proper source conversation links in markdown format * feat(daily): refactor daily note system with enhanced metadata handling - Introduce validate_filename_component function and export it - Add _INDEX_HIDDEN_METADATA_KEYS to hide conversation metadata from index - Update scan_notes to exclude hidden metadata keys from index rendering - Modify auto_memory to use daily_write tool and manage session frontmatter - Implement session note lookup and renaming based on frontmatter name - Update daily_list to return flattened note metadata including session info - Change daily_write to dispatch write step instead of running job - Add test cases for updated daily note functionality and metadata handling - Update version from 0.4.0.2 to 0.4.0.3 * fix(evolve): correct metadata update in auto memory response - Fixed trailing comma issue in metadata dictionary update - Ensured proper formatting of response metadata structure - Maintained existing functionality while fixing syntax error * refactor(auto_resource): replace daily_create with dynamic note management - Remove DailyCreateStep and related exports from file_io module - Replace static daily note creation with dynamic resource-linked card system - Implement LLM-suggested naming with frontmatter-driven file management - Add source_resource linking for tracking original files - Introduce collision handling with hash-based suffixes - Update documentation to reflect new resource card workflow - Modify auto_resource prompts to use write/edit tools instead of daily_create - Adjust test fixture comments to match new agent behavior - Update framework diagrams and quick start examples accordingly * feat(app): add version info to app initialization and update auto-memory logic - Include version number in application startup logging - Remove tool result truncation logic from auto-memory step - Update auto-memory to exclude tool_result blocks from saved history - Add test case to verify tool results are filtered out from message saving - Update YAML prompts to clarify filename naming rules without dates - Modify configuration to support new dispatch steps format with persistence control * feat(auto_memory): add note modification tracking and optimize frontmatter updates - Add _note_bytes and _note_modified methods to track actual file changes - Optimize frontmatter updates by checking existing metadata before update - Add modified flag to response metadata indicating actual note changes - Update logging to include modified status in various operations - Add comprehensive tests for modified/unmodified detection scenarios - Enhance result hook logic to skip when no actual changes occur - Refactor metadata handling to properly track creation vs modification status	2026-06-25 21:54:56 +08:00
jinliyl	8b82ff88d0	feat(evolve): enhance agent reply processing and logging capabilities (#293) ... * feat(evolve): enhance agent reply processing and logging capabilities - Add agent_reply_result_text function to extract final user-visible text from agent replies - Implement comprehensive logging throughout auto_memory, auto_resource, and dream modules - Add max_units configuration option to limit extracted memory units - Improve error handling and validation in auto_resource step - Refactor dream extract step to respect max_units limit during processing - Enhance summary rendering in dream finish step with detailed breakdown - Add result hook functionality for embedding hosts integration - Implement loose resource filename handling for root-level resources - Update test cases to reflect new functionality and improved error messages * test(background-steps): update fake upsert function to include created parameter - Modified fake_upsert function to accept 'created' parameter instead of '_created' - Added 'created' field to captured dictionary in fake_upsert function - Included 'created': True in the expected response dictionary for test case - Updated test assertion to match new parameter structure	2026-06-24 22:08:47 +08:00
jinliyl	a3bd81bde2	Update version to 0.4.0.2 and improve tokenizer index handling (#290) ... * fix(core): update version number to 0.4.0.1 - Incremented version from 0.4.0.0 to 0.4.0.1 in __init__.py * fix(index): remove stopwords path from tokenizer config and add keyword index repair - Remove stopwords_path from tokenizer config to prevent index forking by install path - Add _sync_keyword_index_from_chunks method to repair keyword index when persisted state mismatches - Implement test for keyword index repair from persisted chunks when missing - Add test to verify tokenizer fingerprint ignores stopwords absolute path - Update version from 0.4.0.1 to 0.4.0.2 * feat(dream): add scan_days parameter to dream extraction process - Add scan_days configuration option to default.yaml with default value of 2 - Implement recent_dates utility function to calculate date ranges for scanning - Modify DreamExtractStep to scan multiple days based on scan_days parameter - Update dream extraction to process files across multiple dates instead of single day - Extend DreamState schema to include dates and scan_days fields - Update DreamTopicsStep to handle multi-day topic processing - Modify finish step to checkpoint files from all scanned dates - Add comprehensive tests for multi-day scanning functionality - Update prompt templates to include scan dates information - Refactor topics writing logic to target specific date rather than current date	2026-06-24 15:01:07 +08:00