* feat(file_store): add ZvecLocalFileStore backend
- Implement ZvecLocalFileStore with native zvec collection for ANN search.
- Keep JSONL chunks as the source of truth; rebuild collection from chunks
when sidecar digest/dimension/HNSW M mismatch is detected.
- Add dedicated unit tests in tests/unit/test_zvec_file_store.py.
- Parametrize existing file_store consistency tests to cover both
LocalFileStore and ZvecLocalFileStore.
- Register the new backend in reme/components/file_store/__init__.py.
* fix(file_store): fix zvec collection sync and content validation, declare zvec dependency
* chore(benchmark): isolate dataset/workspaces/results per benchmark
- Move shared benchmark/{datasets,memory_workspaces,results} into per-benchmark subdirs benchmark/<name>/{dataset,workspaces,results}
- Update beam/longmemeval config.yaml and run.py path defaults
- Relocate longmemeval download.py to benchmark/longmemeval/ (downloads into dataset/ subdir); inline dataset download docs into README
- Update .gitignore: benchmark/*/{dataset,workspaces,results}/
- Move result-{beam,longmemeval}.md to benchmark/results_md/ and drop result- prefix; update README links
- Fix stale path refs in llm_judge.py and logs/demo_search_format.py
* feat(benchmark): add read tool to agentic answer and update BEAM results
- Add 'read' to job_tools in BaseAgenticAnswerStep for file reading capability
- Document read tool usage in lme/agentic_answer.yaml system prompt
- Update result-beam.md with latest evaluation scores (OVERALL: 0.623/0.580)
* feat(auto_memory): add source line-number markers for note traceability
- Add _format_history hook in AutoMemoryStep with line-number annotation
- Override in BeamAutoMemoryStep to prefix each turn with [Ln] for citation
- Add session_file variable to prompt templates for source marker paths
- Simplify repeated extraction rules by referencing system prompt
- Enhance agentic_answer search strategy (multi-search, read tool hint)
- Add warning log on ReadStep failure
* feat(beam): enhance auto_memory with source markers and pilot ingest tooling
* refactor(beam): rename max_chunk_words to max_segment_words, drop one-off pilot scripts
* feat: add CompressorStep and search_v2 dual-mode session compression
- Add CompressorStep (reme/steps/evolve/compressor.py) for direct LLM
text compression with optional query-guided relevance filtering
- Extend search_v2_step to support query-aware and query-independent
session transcript compression via _compress injected kwargs
- Refactor _source_format.py: split into render_chunk_entries +
join_chunk_entries; session chunks now render line-aligned with
L<n>: prefixes for verbatim/compressed parity
- Add JOB_TOOLS and INJECTED_JOB_KWARGS to BaseAgenticAnswerStep for
per-subclass tool and parameter injection
- LmeAgenticAnswerStep injects _search._compress payload to enable
query-aware compression during benchmark evaluation
- Record compression ablation results in result-longmemeval.md
- Add unit tests for CompressorStep and search compression paths
* refactor(compress): relax session compression to lenient format-preserving strategy and update LME results
* refactor(benchmark): make session compression config-driven via compress_session flag
Move session-transcript compression from LME hard-coded injection to a
runtime context flag set by evaluation.compress_session in each
benchmark config. Compression is off by default for both BEAM and LME,
and BaseAgenticAnswerStep now conditionally injects the _search compress
payload only when the flag is truthy.
* feat(lme/auto_memory): add source attribution markers with line numbers
Add _format_history to annotate each turn with [Ln] line numbers and
expose {session_file} in prompts so the agent can emit bare wikilink-style
source markers like [[session/dialog/s1.jsonl#L1-L2,L5-L6]] at the end
of factual entries. Consolidate the per-prompt body/format rules into
references to the system prompt to avoid drift, and add frontmatter-
protection guidance for the edit tool.
* feat: improve agentic answer prompt and update beam 100K results
- Strengthen abstention rule: prohibit extrapolation from related but
non-direct evidence
- Add multi-angle search after preliminary answer to check for
conflicting/supplementary/updated information
- Add max-iteration fallback to 'Information not found'
- Update beam.md with 100K results (agentscope 2.0.4.post1, from scratch)
including per-type token consumption and memory construction stats
- config.yaml: 100K dataset, 20 workers for BEAM evaluation
- run.py: add memory construction token usage tracking (default agent)
- Overall: 0.635 → 0.654 (+0.019), contradiction_resolution: 0.338 → 0.478
(+0.140), abstention: 0.500 → 0.525 (+0.025)
* feat(read): add session-aware formatting for read tool and update BEAM eval
- Add truncate_session_output in _file_io.py to render jsonl session
lines as [speaker @ time] content before byte-budget truncation
- Add read_step_format_session flag to ReadStep, honoring injected
job kwargs (precedence) and YAML fallback
- Inject read_step_format_session=True into BaseAgenticAnswerStep
so agentic answer reads render session transcripts human-readably
- Refine BEAM agentic_answer prompt: continue multi-angle search
after preliminary answer, forbid fabrication/extrapolation
- Update BEAM config to 1M variant and add sequential 100K-eval /
1M-build shell script
- Refresh benchmark/results_md/beam.md with latest results
* chore(config): disable expand_links in beam and lme search_v2 configs
* refactor(beam): drop one-off sequential 100K-eval-then-1M-build script
* fix(benchmark): add compressor job to beam config and fix BEAM clone instructions
- Add compressor job and compressor as_llm component to reme/config/beam.yaml
(aligned with lme.yaml) so that compress_session: true works for BEAM
- Add graceful degradation guard in search_v2._compress_session_entries:
when the compressor job is missing from the active config, log a warning
and skip compression instead of raising 'Job compressor not found'.
Skipped when there is no app_context so unit tests mocking run_job still
drive compression behavior.
- Fix BEAM download instructions in README.md/README_ZH.md: add mkdir -p
before cd benchmark/beam/dataset (the directory is gitignored and absent
in a fresh clone)
* fix(steps): guard compressor exceptions and fix ReadStep boolean override
1. search_v2: catch per-entry exceptions from run_job('compressor') inside
compress() so asyncio.gather never propagates a compressor failure (e.g.
temporary LLM outage). The failing entry keeps its original body while
remaining entries are still compressed, preserving already-retrieved
search results.
2. read: replace 'context_value or yaml_value' with an existence check so
that a runtime-injected False can explicitly disable a YAML-true
read_step_format_session flag.
Add focused unit tests for both paths.
* fix(search_v2): use existence check for strict_date_filter boolean override
Replace 'context_value or yaml_value' with an existence-based check so
that a runtime-injected False can explicitly disable a YAML-true
strict_date_filter flag, consistent with the read_step_format_session fix.
* refactor(search): simplify strict_date_filter fallback to truthiness-or
* style(test): rename unused param to satisfy pylint W0613
---------
Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
Make "Experience-driven enhancement method" point to the archived benchmark page while keeping the arXiv link.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(benchmark): archive ExpG tool-use results under toolmemory
Add ToolMemory benchmark materials and link them from the root and
benchmark READMEs so ReMe documents the ExpG tool-use enhancement work.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs: point ToolMemory news entry directly to the paper
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(benchmark): address ToolMemory review and pre-commit
Restore benchmark index READMEs, link ExpG to WangCan1178/ExpG instead
of ReMe version notes, and format tool_memory.py for CI hooks.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(benchmark): align ToolMemory client with official ReMe APIs
Drop ExpG-only request fields and non-official metadata handling so the archived client matches add/summary/retrieve Tool Memory endpoints.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(benchmark): fix trailing whitespace in ToolMemory READMEs
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: simplify local links and support line anchors
* fix: align line anchor tests with CI lint
* fix: preserve local links across file moves
* fix: encode markdown paths when rewriting links
* refactor(read): keep explicit line range parameters
* fix: simplify legacy link predicate compatibility
* docs: align local link behavior with implementation
* fix: skip unsupported markdown destination escapes
* fix: normalize workspace link paths across platforms
* fix: bound markdown link scanning
* fix: keep local link processing linear
* docs: clarify permissive markdown link parsing
* fix: handle local link processing failures
* refactor: limit file links to wikilink syntax
* docs: align wikilink contract with implementation
* fix: normalize dream and neighbor paths on Windows
* fix: resolve workspace path for neighbor expansion
Add the meta-reme subsystem that generates minimal, benchmark-specific
ReMe source bundles from an allowlist. The bundle_builder.py reads
build_bundle.yaml to produce three targets (default, lme, beam), each
validated by import/step-registration checks before being staged.
Also include the design document describing the search-driven harness
optimization loop and a sandbox integration test exercising the first
LongMemEval and BEAM 100K cases against generated bundles.
Add commit_memory_history() so the host controls when daily-memory
changes are checkpointed into the runtime workspace's local Git repo.
Session ingestion no longer commits implicitly; the host groups one or
many sessions into an explicit boundary with a chosen message.
- Initialize a Git repo in the runtime workspace on case setup
- Validate the configured daily_dir and stage only that path
- Allow empty commits so boundaries are always recorded
- Reset removes the old Git repo and reinitializes for the next case
- Export retains .git for offline history inspection
Enhance the Docker benchmark sandbox so one container can process
multiple cases sequentially and export only analysis-relevant artifacts:
- Add reset_case() to discard the active case while keeping the
installed candidate, venv, and worker for the next case.
- Add export(profile="analysis") (default) and export_full() to choose
between a selective analysis archive and the complete disposable case
tree; analysis export omits rebuildable indexes, caches, temp files,
and source resources.
- Introduce runtime-layout.json written by the worker with validated,
non-secret, case-relative paths so the host can build safe tar
excludes.
- Serialize job, export, and reset operations with a per-case lock.
- Pin a sandbox-runtime extra in pyproject.toml that omits cookbook-only
tushare, dingtalk-stream, pypdf, and polars, and install it in
Dockerfile.base.
- Set TMPDIR to a case-local directory so subprocess temp files stay
inside the disposable case root.
- Add unit tests for the new behaviors and a real-Docker LongMemEval
integration smoke test.
- Add version_tuple utility for semantic version comparison
- AsAgentWrapper: version-aware InjectionConfig, max_iters doubling,
and token usage collection (reply vs reply_stream) for AS>=2.0.5/<2.0.5
- Default inject_runtime_state=False in wrapper to avoid benchmark
time-anchor conflicts; remove per-callsite injection_config overrides
- longmemeval run.py: support question_ids filter in dataset config
- Fix unused import in test_evaluation_interface; format fixes
- Add InjectionConfig passthrough in AsAgentWrapper.reply()
- Disable inject_runtime_state in BaseAgenticAnswerStep to avoid
wall-clock time conflicting with benchmark query_time anchors
- Disable inject_runtime_state in beam/lme llm_judge calls
- Extract _counter_key from BaseJob._record_call for reusable counter lookup
- Add reme.utils.evaluation_interface.check_job_count read-only helper
- Track and report average search calls per query in beam and longmemeval benchmarks
- replace global_counter_next with fetch-and-add style global_counter_add/inc, plus read-only global_counter_get and global_counter_get_all
- record per-job call counts in app_context.metadata via BaseJob._record_call, covering background/cron/stream jobs
- update agentic_answer step and utils exports; add unit tests for job counting and counter utils
- Introduce tolerant AutoFinAgentModel base class allowing extra fields in raw Agent outputs
- Replace strict models with tolerant ones for ETF, historical event, market selection, and report outputs
- Remove redundant field validators and allow empty defaults for key string fields
- Enhance historical source path resolution to safely filter invalid or out-of-workspace paths
- Add normalization of whitespace and validation to historical event references before processing
- Implement normalization in Topic and Market Agent selections to eliminate duplicates, blanks, unknowns
- Limit Topic Agent output to top 20 ETFs and ensure sorting and deduplication of events
- Normalize final Markdown report by removing redundant headers and providing safe fallbacks
- Update agent prompts to clarify task constraints and improve instruction consistency
- Add extensive tests for normalization, filtering, and safe source resolution for historical events
* feat(file_store): upgrade FAISS to HNSW index with async reindex and path constraint
- Replace IndexFlatIP with IndexHNSWFlat for better recall/speed tradeoff
- Add dynamic efSearch (limit * 5) scaled to query request size
- Add async_reindex option: background rebuild with generation-based invalidation
- Extract _delete_nodes() in LocalFileStore for subclass reuse
- Add unit tests for file store consistency
* fix: resolve pylint warnings in faiss store and test file
* refactor(file_store): replace generation-based reindex with event-flag worker
- Replace _reindex_generation/lock/task with a single long-lived worker
coroutine consuming an asyncio.Event flag; repeated submissions coalesce
- Use local index reference in vector_search to avoid TOCTOU on self._faiss_index
- Pass index explicitly to _set_ef_search for consistency
- Track _index_writes to re-arm reindex after concurrent writes
- Update tests to match new internal API
* fix: resolve pylint too-many-return-statements and implicit-booleaness warnings
* feat(file_store): add refine maintenance hook and incremental embedding backfill
- Add refine() idle-time maintenance hook to BaseFileStore/LocalFileStore
- FaissLocalFileStore: incremental vector add on backfill instead of full rebuild
- Dynamic tombstone compaction threshold scaled by index size
- Add RefineStoreStep with daily cron job (refine_store_cron)
- Enable faiss backend and embedding_store by default in default.yaml
- Add unit tests for faiss index maintenance
* chore(deps): promote faiss-cpu to core dependencies
faiss backend is now the default file_store, so faiss-cpu moves from
the optional [core] extra to the base dependencies list.
* feat: rename refine_store to optimize_index and add vecdb_path_constraint
- Rename refine_store step to optimize_index with cron job scheduling
- Add vecdb_path_constraint to file_store components
- Update default.yaml with optimize_index_cron and faiss backend comment
- Update memory_search docs (en/zh) for FAISS vector management
- Update unit tests for index maintenance
* feat(faiss): add embedding digest to reject stale sidecar after partial dump
Add _chunks_embedding_digest() that computes an order-independent SHA-256
over (chunk_id, float16 embedding) pairs. The digest is written into the
idmap sidecar at dump time and verified at load time. A mismatch means the
sidecar vectors belong to a different chunk generation than the authoritative
JSONL — detectable even when the live-ID set is unchanged (same-ID in-place
update crash window).
Add test_faiss_rejects_stale_sidecar_after_partial_dump reproducing the
crash-between-writes scenario and asserting digest-based rejection.
Compress verbose docstrings/comments in existing tests for pylint line
budget.
---------
Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
- Add AutoFinHistoricalDirectionReference model to classify historical events by direction
- Remove AutoFinHistoricalSimilarity and related similarity score usage
- Update AutoFinMarketSelection to handle same and opposite direction event lists
- Adjust AutoFinMarketStep to calculate forecasts based on equal weights and direction signs
- Change market.yaml instructions to require direction classification instead of similarity scoring
- Modify tests to reflect direction-based classification and verify uniqueness across direction groups
- Improve DingTalkWaitStep to support reconnect on server request with proper disconnect reason handling
content.split("\n") yields a trailing empty element for any file ending in a
newline, inflating total by 1 and letting a start_line one past real EOF be
silently accepted instead of rejected. Mirrors the trailing-newline correction
default_file_chunker already applies, and matches the large-file path's
line-by-line count.
Fixes#388
- Separate candidate source file resolution from event resolution logic
- Allow fallback to date-derived daily news file if original source is missing
- Check existence and validity of historical source files more robustly
- Handle multiple candidate source files and aggregate matches before validation
- Gather and log resolution limitations without stopping processing
- Return resolved events along with a list of resolution warnings
- Update related code to consume new return signature and merge limitations
- Add detailed validation on source path relativity and file naming conventions
* feat: add ssh proxy
* feat: add ssh proxy
* feat: add ssh proxy
* feat: add ssh proxy
* feat: add prompt
* feat: add agent wrapper
* feat: add agent wrapper
* feat: add agent wrapper
* feat: add tushare skill
* feat: add tushare skill
* feat: add tushare skill
* feat: add none stream
* chore(deps): update dependency versions in pyproject.toml
- Bump claude-agent-sdk from 0.2.123 to 0.2.126
- Upgrade pre-commit to version 4.6.1 or higher
- Upgrade pytest to version 9.1.1 or higher
* feat(agent_wrapper): add session compaction support and unify session commands
- Introduce compact_session method to BaseAgentWrapper and implement it in AsAgentWrapper, CcAgentWrapper, and CodexAgentWrapper
- Add session_command module with SessionCommandResult dataclass and handle_session_command function for /clear and /compact commands
- Update __init__.py exports to include session_command handlers
- Modify DingTalkWaitStep to handle session commands via handle_session_command function
- Remove streaming mode from DingTalkWaitStep and simplify reply handling to final Markdown replies only
- Add unit tests for session compaction methods and session command handling across wrappers and DingTalk integration
- Clean up and remove obsolete streaming and card rendering code from DingTalk wait step
- Adjust daily_cookbook.yaml to remove stream and card_update_interval config entries for DingTalk wait step
* feat(auto_fin): add Auto Fin simulated portfolio cookbook workflow
- Add comprehensive Auto Fin schema exports for multiple models and enums
- Implement base class and helpers for Auto Fin analysis steps
- Create file, state, and formatting utilities for Auto Fin with atomic file writes and locking
- Define Auto Fin pipeline with four analysis agents: backtest, event, portfolio, and US correlation
- Register Auto Fin package in cookbook workflows and schema initialization
- Add detailed documentation in markdown describing the system design, workflow, and data contracts
* feat(outbound_proxy): add application-scoped outbound HTTP proxy components
- Introduce BaseOutboundProxy and OutboundProxyEndpoint as core contracts
- Implement FixedHttpOutboundProxy for external HTTP proxy integration
- Add SshHttpOutboundProxy providing SSH-backed local HTTP proxy tunnels
- Register outbound proxy components in component registry and enumeration
- Update components package to include outbound_proxy module
- Add dependency on pproxy for SSH HTTP proxy bridging
- Include comprehensive unit tests covering proxy lifecycle, validation,
environment merging, error handling, readiness, and monitoring mechanisms
* refactor(network): replace SSH proxy with explicit HTTP outbound proxy
- Remove SSH proxy helper implementation and references in codebase
- Add support for explicit HTTP proxy URL in arXiv and HuggingFace clients
- Modify clients to use async context manager for consistent resource handling
- Update daily paper steps to forward outbound proxy configuration explicitly
- Change tests to cover new proxy usage model and remove SSH proxy mocks
- Add outbound proxy component configuration in daily_cookbook.yaml
- Ensure proxy URL usage disables environment trust in HTTP clients
- Fix app context component enum access to be defensive against missing keys
* feat(agent_wrapper): add managed proxy support for command environments
- Introduce BaseOutboundProxy binding in BaseAgentWrapper for outbound proxy management
- Add bash_environment and command_proxy_environment properties to apply proxy settings
- Update WorkspaceBackend instantiation in AsAgentWrapper to use bash_environment
- Inject managed proxy export commands into Claude Code Bash commands via hooks
- Enhance CodexAgentWrapper to include managed proxy in shell environment policy
- Modify daily_cookbook.yaml steps to specify outbound_proxy as default where needed
- Add comprehensive unit tests verifying managed proxy injection and environment isolation
- Ensure subprocess_environment remains unchanged while proxy is applied selectively to commands
* refactor(memory): replace search job_tools with memory in daily cookbook config
- Change workspace_dir default from .reme to reme_workspace
- Replace search job_tools with memory across multiple components and jobs
- Update descriptions to reflect long-term memory retrieval instead of search
- Modify system prompts to instruct using memory for retrieving notes
- Adjust unit tests to verify memory job_tools and job presence instead of search
- Ensure consistency in configuration and tests for memory backend usage
* refactor(config): rename memory to memory_search in daily cookbook config
- Change all occurrences of "memory" to "memory_search" in job_tools and job definitions
- Update related system prompts to reflect the new memory_search terminology
- Modify unit tests to assert the presence of memory_search instead of memory
- Ensure consistency across skills, job tools, and backend configurations in multiple components
* feat(auto_fin): add deterministic quantitative research and ranking fusion
- Introduce new schema models: EtfScore, RankingMetrics, ExtremeAnalysis,
DimensionRanking, and FusionRanking to represent deterministic research outputs
- Add ranking data to event, backtest, us_correlation, and portfolio analysis outputs
- Implement ranking_section renderer to format Top20 scores and diagnostics in Markdown
- Develop AutoFinQuantStep for deterministic ETF ranking using TuShare data, Polars,
and a custom extremely randomized tree ensemble
- Integrate quantitative rankings into backtest and portfolio analysis steps and reports
- Extend auto_fin pipeline with new quant_enabled and quant_required config options
- Enforce ranking constraints like unique codes, contiguous ranks, and normalized fusion weights
- Update analysis YAMLs with rules limiting data freshness, universe, and ranking usage
- Incorporate ranking outputs into all major markdown report bodies in Auto Fin pipeline
- Add concurrency-limited asynchronous TuShare client to fetch required market data
- Introduce cross-sectional rank correlation and NDCG metrics for ranking quality evaluation
* feat(auto_fin): implement stage-wise notification and reporting for analysis pipeline
- Refactor notification config in daily_cookbook.yaml to support dispatch steps
- Update AutoFinNotificationStep to deduplicate notifications per run stage
- Add _notify_stage method in pipeline to send notifications for each analysis stage
- Implement persistence and notification for event, backtest, US correlation, and portfolio stages
- Modify pipeline flow to persist reports and notify after each stage completion
- Adjust metadata to track notifications and errors per stage
- Update tests to verify stage-wise notification sending and deduplication
- Remove older combined report persistence in favor of modular stage handling
* feat(auto_fin): add outbound proxy support for Tushare API usage
- Introduce BaseOutboundProxy reference in AutoFinPipelineStep and AutoFinQuantStep
- Update TushareResearchClient and trade calendar fetch to accept and use proxy URL
- Create _ProxiedTushareApi adapter to route Tushare requests via explicit HTTP proxy
- Modify create_tushare_api utility to optionally return proxied API client
- Add unit tests covering proxy forwarding and client behavior with managed proxies
- Ensure proxy usage respects explicit proxy URL over environment fallback
- Integrate outbound proxy into data fetching and quantitative research steps
* feat(auto_fin): enforce checkpoint time validation and add state models
- Introduce AnalysisState base class and specific states for event, backtest, and US correlation analyses
- Replace analysis output types with corresponding state classes in run schemas
- Add require_checkpoint_reached method to validate decision_at/data_cutoff against current time
- Enforce checkpoint time checks before analysis steps in event, backtest, portfolio, and quant analyses
- Refactor quant data loading to include adjustment factors and apply price adjustments without fallback
- Update analysis YAML docs to require real-time checkpoint validation and forbid using future data
- Improve portfolio run serialization by excluding redundant legacy fields and nested proposed actions
- Add helper to extract readable sections from persisted checkpoint documents
- Fix event analysis output validation to reject events and sources with future timestamps
* feat(auto_fin): auto-select latest reached checkpoint if none specified
- Extend checkpoint config to accept empty string for auto selection
- Add static method to compute latest checkpoint reached by current time
- Modify pipeline step to auto-select checkpoint based on trade calendar and time
- Adjust force flag default depending on whether checkpoint is explicit or auto
- Log details when checkpoint is auto-selected to improve observability
- Add comprehensive tests for auto checkpoint selection logic and edge cases
- Remove deprecated default and required constraints from force parameter in config
* refactor(auto_fin): unify datetime comparison with compare_datetimes utility
- Replace direct datetime comparisons with compare_datetimes function calls
- Use cmp_to_key with compare_datetimes for sorting datetime tuples and lists
- Update validation logic in backtest, event, analysis, and ledger modules for consistent datetime handling
- Add unit tests to verify handling of naive and aware datetime comparisons in event and backtest validations
- Ensure marked_at and interval_end timestamps are set and compared consistently using compare_datetimes
- Improve correctness of ordering and conditional checks related to timestamps throughout auto_fin steps and ledger code
* feat(auto_fin): add datetime comparison helper for mixed timezone data
- Implement compare_datetimes function to handle naive and aware datetimes
- Ensure naive datetime is interpreted in the known timezone of the counterpart
- Facilitate comparisons between legacy and timezone-aware Auto Fin data
- Add module docstring explaining purpose of the helpers
* docs(auto_fin): enforce unique ETF representative per sub-theme in analysis rules
- Update backtest.yaml to recommend or highlight only one ETF per sub-theme for ETF analyses
- Modify event.yaml to map only one representative ETF per sub-theme, avoiding duplicate recommendations
- Revise portfolio.yaml to restrict holdings/buys to a single ETF per sub-theme, preventing repeated buys of highly overlapping ETFs
- Adjust us_correlation.yaml to retain only one representative A-share ETF per sub-theme for mapping or recommendation
- Add test to verify presence of new sub-theme uniqueness guidance in step prompts
* feat(auto_fin): separate draft model and include deterministic fusion ranking
- Introduce _PortfolioProposalDraft pydantic model for agent-authored fields before ranking
- Discard any "fusion_ranking" data from draft to prevent conflicts with canonical ranking
- Modify AutoFinPortfolioStep to receive draft, enrich with fusion_ranking, and produce final output
- Update tests to use _PortfolioProposalDraft and validate deterministic fusion ranking propagation
- Add async test verifying fusion ranking is correctly set in portfolio output with no errors
* refactor(auto_fin): rewrite and simplify Auto Fin schema and steps
- Remove legacy Auto Fin analysis step modules and helpers
- Replace complex ranking and portfolio models with simplified current-news models
- Update schema to focus on news-case workflow with new domain models
- Remove A-share decision checkpoints and backtest details from schema
- Simplify recommendation and decision output structures
- Clean up deprecated state and utility functions
- Update Auto Fin steps initialization to new pipeline steps only
- Improve uniqueness validation for themes and ETFs in research plan
* feat(auto_fin): implement full local cache and analysis workflow for Auto Fin
- Add AutoFinDataStep to prepare and cache daily TuShare data with lookback
- Add AutoFinAnalysisStep to analyze cached data and generate Markdown report
- Implement detailed time window, ETF filtering, and historical case validation
- Introduce YAML prompts for planning and decision-making steps
- Update .gitignore to include reme_workspace/
- Clean up config and import structure for auto_fin steps
- Remove old pipeline.py and consolidate functionality into new modules
- Use polars for efficient CSV reading and data processing
- Ensure atomic writes and strict JSON serialization for cache files
- Enforce rules on news timing, ETF universe, and historical case usage
* fix(auto_fin): restrict news data source to '财联社' in analysis and cache
- Update analysis templates to specify current news as from '财联社' only
- Modify news fetching functions to filter by source '财联社'
- Add validation method to check cached news source correctness
- Update news caching logic to exclude non-'财联社' news
- Enhance unit tests with multiple sources to ensure filtering works
- Confirm news API calls include source filter parameter as '财联社'
* refactor(auto_fin): convert I/O methods to asynchronous implementations
- Change _news, _dataset, and _theme_data methods to async for improved concurrency
- Move JSONL and CSV reading operations to asynchronous wrappers using asyncio.to_thread
- Remove synchronous _read_jsonl and _read_csv functions, integrate them as static async class methods
- Update cache validation methods to async, awaiting I/O operations accordingly
- Adjust usage of dataset and news retrieval in analysis step to await asynchronous methods
- Add async unit test to validate JSONL reading with unicode line separators
- Preserve existing functionality while enabling non-blocking file and data access
* fix(nx_file_graph): defer networkx import and improve dependency handling
- Move networkx import inside NxFileGraph constructor for lazy loading
- Raise ImportError with original exception context if networkx is missing
- Remove module-level fallback assignment of nx to None
- Expand test to block loading of multiple optional core dependencies eagerly
- Change exception type in test from ModuleNotFoundError to AssertionError
- Update test comments to reflect broader optional dependency checks
* feat(embedding_store): add quota retry delay mechanism for embedding requests
- Introduce quota_retry_delay parameter to configure wait time before retry on quota exhaustion
- Implement detection of insufficient quota errors in LocalEmbeddingStore without external SDK
- Add retry logic with custom delay when quota is insufficient during embedding requests
- Update configuration to set max_retries and quota_retry_delay defaults for embedding store
- Add unit tests covering quota exhaustion retry behavior with delay and opt-in control
- Ensure existing retry behavior remains unchanged if quota_retry_delay is not set
* feat(auto_fin): add detailed logging to analysis and data fetching steps
- Add _preview static method for bounded diagnostic output in analysis.py
- Log prompt start, completion, errors, and validation details in _reply method
- Add info logs for major processing steps in execute method of analysis.py
- Add debug and info logs for cache validation, data fetching, and pagination in data.py
- Log conditions for skipping reports and cache plans in data.py execute method
- Log download summaries and cache writes for news and ETF data
- Improve error logging with exception details in cache validation functions
- Ensure all logs include context such as record counts, paths, and parameters
* refactor(auto_fin): overhaul Auto Fin workflow and schema contracts
- Replace old Auto Fin schema models with comprehensive new data classes
- Remove legacy Auto Fin analysis step in favor of modular agent-based steps
- Introduce AutoFinAgentStep for validating structured agent replies
- Simplify data cleaning and JSONL writing utilities for news cache
- Remove synchronous and asynchronous dataset methods from analysis step
- Redefine Auto Fin analysis configuration for 360-day news retention and multi-step pipeline
- Remove embedded analysis prompt templates and replace with agent-driven logic
- Update __init__.py exports to match new step implementations and remove deprecated classes
- Improve error handling and validation in agent step reply processing
- Clean up redundant imports and unused code in analysis and data preparation modules
* feat(auto_fin): add detailed logging for analysis and data processing steps
- Add timing logs to measure agent prompt processing duration in analysis.py
- Log news cache hits and news write paths with record counts in data.py
- Include detailed info logs for news download start and completion in data.py
- Add start, progress, and completion logs with topic and event counts in history.py
- Log start and completion of merge step including path and ETF count in merge.py
- Add start and done logs with window and news counts in topic.py
* feat(auto_fin): enhance schema and steps with detailed ETF and event modeling
- Replace and add multiple AutoFin schema classes to support detailed ETF selection,
historical research, market analysis, forecast models, and report output with validation
- Implement Shanghai timezone normalization and strict validation in schema models
- Remove deprecated AutoFin analysis agent step and consolidate reply handling in base step
- Introduce AutoFinStep base class with shared helpers for prompt handling, data fetching,
logging, and JSONL file operations
- Add AutoFinDataStep to manage daily news data complete with schedule validation, caching,
and source validation logic
- Update cookbook configuration to customize auto_fin step parameters and simplify
outbound proxy settings
- Refactor imports and clean unused code for better maintainability
* feat(auto_fin): introduce detailed historical event resolution and market similarity analysis
- Add AutoFinHistoricalEventReference and AutoFinHistoricalSimilarity models for refined event referencing and similarity judgment
- Implement validation to ensure non-empty critical fields and uniqueness of historical news IDs
- Develop method to resolve Agent-selected historical event references from workspace files with strict path and existence checks
- Enrich historical events with market entry and future returns data after resolution
- Redesign market step to calculate similarity-weighted ETF forecasts based on matched historical event similarities
- Enforce validation on matched historical events for uniqueness and proper weight summation
- Simplify merge step output to final Markdown report without YAML frontmatter and redundant fields
- Update user instructions for history search, market, and merge steps to reflect new data structures and responsibilities
- Adjust test suite to cover new schema and step behavior changes, including enhanced validation and JSON output formats
* feat(auto_fin): add new cron jobs and output analysis jsonl
- Add new cron jobs auto_fin_1145_cron and auto_fin_1800_cron with auto_fin_steps
- Change auto_fin_0930_cron schedule to run Monday to Sunday
- Extend merge step to write analysis data to auto_fin_analysis.jsonl
- Update unit tests to verify new cron jobs and their steps configuration
* fix(auto_fin): improve atomic file write and refresh daily index
- Change temporary file naming to include UUID for uniqueness and hidden prefix
- Replace atomic write method from using Path.replace to os.replace with safe unlink
- Add import and use os.replace for safer file replace operation
- Refresh daily index after writing auto finance markdown and JSONL files
- Import and call refresh_day_index in merge step to update file index asynchronously
* docs(cookbook): add optional SSH proxy configuration in README files
- Introduce optional SSH proxy setup in auto-fin and daily_paper cookbooks
- Provide instructions to enable outbound proxy via `daily_cookbook.yaml` and environment variables
- Add `REME_PROXY_IP` and `REME_PROXY_ACCOUNT` environment variables descriptions in multiple README files
- Update English and Chinese README and README_ZH documents with proxy details
- Maintain consistent formatting of environment variable tables across documents
* fix(file_io): include schema_version in hidden metadata keys
- Added "schema_version" to _INDEX_HIDDEN_METADATA_KEYS in _daily_index.py
- Updated _render_notes_block to always include additional keys regardless of schema_version
fix(deps): move pproxy dependency to later in pyproject.toml
- Removed pproxy from early dependencies list
- Added pproxy back near the end of dependency list for better ordering
fix(outbound_proxy): require pproxy package for ssh_http proxy
- Added importlib.util check for pproxy package presence
- Raise RuntimeError if pproxy is not installed when using SSH HTTP outbound proxy
- Improved error message suggests installing reme-ai with 'core' extra
* docs(readme): update News section with new Cookbook workflows
- Clarify introduction of optional Cookbooks with Daily Paper and Auto Fin workflows
- Update English README to reflect both paper discovery and file-native ETF event research
- Revise Chinese README to include financial news and historical market data research capability
- Maintain announcement of paper acceptance at Findings of ACL 2026
* feat(auto_fin): add calculation results to final Markdown output
- Implement _calculation_results to summarize forecast for each ETF analyzed
- Include program-calculated results in the JSON input for the Markdown report
- Update YAML template to incorporate calculation results and adjust recommendation rules
- Refine recommendation logic to rely on event impact judgments combined with calculation outputs
- Modify tests to verify presence of calculation results and updated report content and format
* up prompt
* fix(keyword_index): ignore non-indexable chunks during keyword sync
- Add is_indexable method to base and BM25 keyword index classes to check text tokenizability
- Update local file store to exclude non-indexable chunks from expected document IDs to prevent rebuild
- Fix JSONL chunker to correctly handle Unicode line separator U+2028 inside JSON strings without splitting
- Add test to ensure non-empty but non-indexable chunk does not trigger keyword index rebuild
- Add test to verify U+2028 character does not cause incorrect JSONL record splitting
