* fix(reindex): return a readable summary so Studio stops showing a placeholder
`ReindexStep` assigned the store's report dict straight to
`response.answer`. ReMe Studio's settings panel already anticipates a string
answer:
typeof result.answer === "string" && result.answer ? result.answer : t("indexRebuilt")
(`reme_studio/app/settings-center.tsx`), so with a dict answer that branch was
unreachable and every rebuild reported the generic "index rebuilt" text. The
operator never saw which scopes ran or how many entries each indexed.
Render the report as one summary line — `Reindexed scope=all: bm25=7,
embedding=5, tag=2` for the composite scope and `Reindexed scope=bm25: 3
indexed` for a single one. The report itself still lands in metadata, so
programmatic consumers are unaffected, and Studio needs no change because its
string branch now fires.
`reindex` is the only answer-carrying consumer of this step; no test asserted
the answer content, only metadata, so nothing else moved.
* chore(reindex): tighten the renderer and deduplicate the summary test
Review on #565: the helper's docstring carried the Studio background
that already lives in the PR description; the intermediate per_scope
dict collapses into the join generator; and the all-scope delegation
test now uses distinct counts (7, 5, 2) so it alone pins both the exact
summary answer and the full metadata report, replacing the overlapping
summary test.
* refactor(reindex): share summary return formatting
---------
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
benchmark.yaml's python_execute job advertised a `timeout` parameter, but
`timeout` is one of the CLI's client-selection kwargs (reme/reme.py pops it
before the payload is sent), so `reme python_execute code=... timeout=300`
never delivered the timeout to the job. The value silently became the HTTP
client's request timeout while the subprocess kept the 60s default, so
benchmark code that needs longer was killed at 60s with no hint why. HTTP
and MCP callers were unaffected.
Rename the parameter to `python_timeout`, mirroring how the shell job
already avoids the same collision with `shell_timeout`. The step prefers
the new name and still accepts the legacy `timeout` context key so
existing benchmark configs keep working. A new guard test fails if any
job in reme/config/*.yaml or a plugin plugin.yaml declares a
client-reserved parameter name again.
* refactor(plugins): move BaseAgenticAnswerStep into lme/beam plugins
Move the shared benchmark answer base class from the core
reme.steps.benchmark package into each plugin's own src tree
(reme_lme.base_agentic_answer / reme_beam.base_agentic_answer) with
absolute imports, drop the core benchmark steps package, and update
plugin READMEs accordingly.
* refactor(plugins): simplify benchmark answer steps
* docs(plugins): clarify benchmark answer step migration
---------
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
* fix(auto-fin): parse topic IDs from fenced JSON replies
* fix(auto-fin): limit report agent tool calls in prompt
* refactor(auto-fin): research news by topic before market open
* fix(config): update default model version for claude_code backend
- Change model version from qwen3.8-max to qwen3.7-plus
- Use environment variable LLM_MODEL_NAME to allow override
- Ensure backend configuration reflects updated model setting
* refactor(auto-fin): write one note per topic before the daily digest
The merge step did two jobs at once: it researched every topic and
combined the results into a single report. Split it the way daily-paper
separates analysis from its brief, so each topic earns a durable note of
its own.
- auto_fin_research_step writes one note per topic that had relevant
news, tagged `kind: auto-fin-topic` and `topic` in frontmatter
- auto_fin_digest_step merges those notes into the day's brief with no
tools of its own and appends a `## 主题详解` section linking back to
each note
- a same-day rerun finds a topic's note by its `topic` frontmatter and
replaces it in place, deleting the old file when the title changed
- base.py now owns the shared Markdown layer: title sanitizing, report
normalizing, wikilink validation, note lookup, atomic frontmatter
writes, and change tracking, so both steps share one write path
- the DingTalk step maps `auto_fin_digest_path` to `markdown_path`
explicitly instead of relying on whichever step ran last
- drop the unused AutoFinTopicOutput schema and read `job_tools` from
the step config rather than hardcoding `search`
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(dingtalk): surface rejection details when delivery fails
A failed group send only reported HTTPStatusError, so an operator had to
reproduce the request by hand to learn why DingTalk refused it. Include
the status code and the whitelisted error keys from the response body in
both the log line and the raised RuntimeError.
Only `code`, `message`, and `requestid` are reported: the request body
carries the message content and credentials, so an error response that
echoes it back must not reach the log. Detail is truncated to 200 chars.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(test): make the suite green on CI
- Point the cookbook claude_code model assertion at qwen3.7-plus, the
default commit 9404e600 set, so the pre-existing red stops blocking
- Satisfy pylint on the auto-fin tests: prefer implicit booleaness for
the recorded Agent calls and drop an unused tmp_path fixture
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor(auto-fin): name notes after the topic, not the Agent title
The research Agent returned a whole paragraph as its title; that became a
filename and blew past the filesystem's 255-byte name limit, failing with
ENAMETOOLONG inside resolve_note_path. Topics are configured values, so
they are short and predictable - use them for file names and keep the
Agent title in frontmatter.
- Name topic notes after the topic and the digest after the run date
- Fold a byte budget into normalize_title as a safety net for long topics
- Take an AutoFinReportOutput in _write_report instead of loose fields
- Ask both prompts for a short title now that it is display-only
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: isolate per-topic research failures and scope frontmatter reads
A single failing topic used to fail the whole cron job and discard the news
already gathered for the topics that had not run yet -- the 09-19 09:24 run
lost its robot notes that way. Research now logs the failure, continues with
the remaining topics, and only fails the run when no topic produced a note.
frontmatter_read was the only frontmatter step without the _allowed_paths
check that read, write, edit, and frontmatter_update already honour, so an
Agent scoped to one file could still read another file's metadata.
- Isolate per-topic research failures and report them as failed_topics
- Fail loudly when every topic fails so an empty brief is never sent
- Apply _check_path_permission in FrontmatterReadStep
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(auto-fin): keep hand-edited notes and wikilink delimiters from breaking the run
Addresses three review findings on the topic-per-note rework.
- `find_note` and `read_note` now skip a note whose YAML frontmatter does
not parse. A hand-edited note in the day directory raised
`yaml.parser.ParserError`, which per-topic isolation surfaced as
"Auto Fin research failed for every topic" and took the run down with it.
- `normalize_title` also strips `[`, `]` and `#`, which `WikilinkHandler`
treats as target delimiters. `AI[算力]` used to emit a trailer link the
parser could not read at all, and `C#` resolved to `.../C` plus an anchor.
- `_write_report` returns the body it actually wrote, and both callers
propagate it, so the digest answer and the note handed to the digest Agent
no longer carry links that validation had already downgraded on disk.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* perf(auto_resource): reuse historical ownership lookup per batch
Share a lazy ownership index across routed processors and expire it at the end of each resource batch. Refresh changed days and their safe aliases after writes, deletes, and partial failures.
Add regression coverage for scan counts, ownership, lookup scope isolation, and retry behavior.
* fix(auto_resource): reconcile shared lookup across active batches
* fix(auto_resource): refresh batch ownership from filesystem metadata
* refactor(auto_resource): simplify batch ownership lookup
Keep per-day ownership caches paired with pre-read metadata and refresh changed days before subsequent resources. Remove reverse-index bookkeeping, post-read stabilization retries, and recursive scope fallback.
Handle late ELOOP errors on Python 3.13 while preserving other OS errors. Update regression tests for cross-resource freshness and retain safety and lifecycle coverage.
* refactor(auto_resource): streamline batch lookup refresh
* refactor(auto_resource): scope ownership cache to batch mutations
Replace filesystem snapshots with one lazy historical lookup shared by resource processors. Reload only days changed by this invocation and discard the cache when it ends.
Reuse canonical daily-note scans for symlink safety and Python 3.13 compatibility. Consolidate regression tests around the narrowed batch contract.
Validation: Python 3.11, 3.12 and 3.13 core/plugin suites each passed 1407 tests; pre-commit --all-files and offline lookup/deletion comparison passed.
* refactor(auto_resource): reuse initial ownership without refreshes
* test(auto_resource): streamline ownership lookup regression coverage
* feat(plugins): auto-tag generated reports
* fix(logging): forward host records on Python 3.13
* refactor(tags): decouple auto tagging from index updates
* fix(tags): bind auto tagging to configured index
* fix(tags): preserve standalone default index
* fix(tags): make auto tagging best effort
* Add optional tag generation and normalization to auto memory
* Add tag index components and clean up temporary JSONL files
* Preserve tag index state when reconciliation fails
* Refactor and streamline application implementation
* Fix pylint C1803 warnings in tag normalization tests
* Document optional tag index configuration
* Make tag index failures non-blocking and disable auto-memory tags
* Add configurable tag indexing and tag listing
* Add tag-filtered hybrid search with exact candidate ranking
* Remove obsolete generated files
* Rename tag index key to tag_key and reject reserved fields
* Extract automatic tagging into a dedicated step
* Restrict frontmatter updates to authorized keys
* Refine tag filtering and automatic memory tagging
* Require underscore-separated tags in auto-tag prompts
- Forbid spaces in tags and require underscores (e.g. sam_altman) in both
English and Chinese auto_tag prompts, with English examples switched to
English entities (OpenAI, gold)
- Drop prompt-string assertions superseded by the new tagging rule
- Merge construction/runtime tag_key validation tests into one parametrized case
* Make max_tags_per_file configurable in auto-tag step
* Consolidate tag index tests
* Fix search test fixture lint warnings
* Align tag contracts and index health behavior
* Fall back when tag index is unavailable
---------
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
* refractor(proactive): upgrade proactive feature with disentangled job and steps
* refactor(proactive): apply audit fixes
- rename read-side job 'proactive' -> 'proactive_read' (less confusing vs the refresh pipeline)
- drop dedicated agent_wrapper.proactive; extraction reuses the default wrapper
- simplify schema: remove unused ProactiveExtractOutput/TopicUpdate, drop resource_paths
- extract no longer scans resource/ directly (daily notes already carry resource content)
- update tests and docs accordingly
* feat(proactive): strict extract-output gate and prompt total budget
- parse_extract_reply now requires a contract section (follow_ups/extends/updates
as a list); non-empty replies with misspelled section names trigger the
existing one-shot retry instead of silently checkpointing changed files
- pack_paths gains max_total_chars; extract packs newest daily material first,
keeps the first file on overflow, and records omitted files in a trailer
(default budget 300000 chars, configurable via max_total_chars)
- tests: schema gate unit, schema-error retry e2e, budget unit + e2e
* feat(proactive): add scenario-card plan step and generative agenda step
* feat(proactive): digest-personal profile personalization and leaner LLM contract
- extract/plan/agenda now draw a user profile block from <digest_dir>/personal/*.md
(frontmatter description + body excerpt, per-file budget, profile.md fallback)
- all daily access honours the configured daily_dir (prompt paths parameterized,
config-driven fallbacks) so workspaces using e.g. memory/ work unchanged
- schema trim: drop dead fields errors/material_paths, carry_forward_all -> count
- shrink LLM output contract: new topics emit title/reason/confidence/paths only;
keywords removed end-to-end, evidence derived from paths[0] (updates keep it)
* fix(proactive): skip checkpoint when extract reply stays unusable after retry
Two consecutive unparseable replies now short-circuit the round without
checkpointing, so the same material is retried next round instead of being
silently consumed (closes the residual audit #1 gap: the structural gate
detected schema-wrong output but a double failure still checkpointed).
* fix(proactive): replace running bool with reference-counted job activity tracker for the idle gate
* refactor(proactive): remove job activity tracking and idle gate, restore job tree to upstream
* fix(proactive): address second audit round (readonly reader, mtime checkpoint, wider fallbacks, profile containment, horizon content, expiry boundary)
* refactor(dream): strip interests.yaml ownership from dream, proactive is now the sole writer
* refactor(dream): separate proactive topic generation
* ci: update renamed auto dream smoke test
* fix(proactive): complete refresh migration and docs
---------
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
* Add optional tag generation and normalization to auto memory
* Add tag index components and clean up temporary JSONL files
* Preserve tag index state when reconciliation fails
* Refactor and streamline application implementation
* Fix pylint C1803 warnings in tag normalization tests
* Document optional tag index configuration
* Make tag index failures non-blocking and disable auto-memory tags
* fix(tag-index): fail closed and support reindexing
* fix(tag-index): preserve complete query expressions
---------
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
* feat: add entry-point plugin system
* fix: harden plugin config and client loading
* docs(workflow): add detailed manual for publishing reme-auto-fin to PyPI
- Provide step-by-step instructions for updating project.version and merging branches
- Explain dependency verification for reme-ai on PyPI during build
- Specify requirements for GitHub Actions secret configuration and version uniqueness
- Describe manual workflow triggering and input of version number
- Recommend publishing order for related projects
- Clarify that only manual dispatch triggers publishing, no automatic triggers on push or tag
* feat: support plugin-defined component types
* refactor: simplify plugin configuration
* fix: isolate plugin loading and defer client fallback
* refactor: freeze built-in component registry
* fix: isolate config entry point loading
* fix: complete auto-fin package metadata
* feat: refine local-first research workflows
* fix: delegate structured output tool choice
* refactor(auto-fin): fetch and filter rolling CLS news
* fix(auto-fin): keep imports portable across platforms
* feat(auto-fin): expose CLS fetch controls
* fix(auto-fin): propagate configurable news window
* feat(auto_fin): normalize hybrid wikilinks in report body
- Add _normalize_hybrid_wikilinks method to remove redundant Markdown destinations
- Use regex to identify hybrid wikilinks with optional destinations
- Replace redundant destinations with simpler wikilink format for clarity
- Ensure normalization is failure-safe with exception handling and logging
- Update report body normalization process to apply hybrid wikilink fix
- Add unit tests to verify correct normalization and failure safety behavior
* fix(dream): serialize integration with application-wide asyncio lock
- Add application-wide asyncio.Lock to serialize digest writes during integration
- Update _snapshot_digest to capture metadata per bucket
- Validate bucket association when recovering from file changes
- Add tests ensuring recovery only from the correct bucket
- Add tests confirming integration lock is shared across application context
- Enhance strict topic YAML loading validation in dream utils
- Add tests for strict topic loading rejecting invalid or lossy fields
* fix(cookbook): enable configurable job_tools for digest and merge steps
- Update daily_cookbook.yaml to add job_tools: [memory_search, read] in digest steps
- Modify DailyPaperDigestStep to read job_tools from kwargs instead of fixed list
- Modify AutoFinMergeStep to similarly read job_tools from kwargs
- Update tests to pass job_tools explicitly when invoking these steps
- Remove hardcoded _TOOLS constants and replace with dynamic job_tools handling
* fix: retry incomplete dream receipts
* perf(pdf): increase max PDF pages limit from 20 to 35
- Updated configuration max_pdf_pages from 20 to 35 in daily_cookbook.yaml
- Modified code to extract up to 35 pages instead of 20 in analyze.py
- Updated README and README_ZH to document the increased max_pdf_pages
- Adjusted unit test assertions to reflect new max_pdf_pages limit of 35
* fix memory integration and daily paper links
* docs clarify cookbook tool usage
* feat(daily-paper): add Hugging Face mirror switch
* refactor(daily-paper): simplify the HF mirror switch and warn on ignored env
The switch was a three-state bool|None where None preserved the legacy
environment-driven selection, but no production caller ever passes None --
collect.py always resolves an explicit bool. Collapse it to a plain bool
defaulting to False.
HF_MIRROR_URL no longer redirects traffic on its own, so warn when it is
configured while the mirror stays disabled; a mirror-only setup would
otherwise fall back to the official site with no signal. Both READMEs now
record the behavior change and stop presenting the two mirror variables as
symmetric -- arXiv remains environment-driven while Hugging Face is gated on
the job parameter.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(daily-paper): address mirror configuration feedback
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* refactor: rebuild auto-fin and daily-paper cookbooks on structured-output agents
Rework the auto-fin and daily-paper cookbooks to run on structured-output
LLM agents instead of Claude Code agent wrappers, replace the SSH proxy with
data-source mirrors, and rewrite the affected unit tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(auto_fin): unify JSON output serialization and writing
- Extracted _write_output static method to serialize and write Pydantic models as compact JSON
- Replaced inline JSON dump and write calls with _write_output usage across auto_fin steps
- Added _report_path and _current_report for managing intra-day reports in AutoFinMergeStep
- Updated auto_fin merge step to write output via new _write_output method
- Enhanced news reading with caching in AutoFinHistoryStep
- Refined returns calculation to handle events before close on non-trading days correctly
feat(daily_paper): improve note path resolution and metadata handling
- Introduced iter_note_metadata generator for safe Markdown frontmatter iteration
- Added resolve_unique_note_path to avoid note filename conflicts on disk and in used titles
- Updated analyze, collect, digest, and select steps to use centralized constants and helpers
- Used utc_now_iso for consistent timestamping in metadata
- Replaced direct frontmatter loads with iter_note_metadata in collect and analyze steps
- Replaced hardcoded paper selection count with PAPER_COUNT constant in all relevant places
- Added _MAX_SELECT_ATTEMPTS constant in select step for attempt management
- Improved error messages for filename validation in daily paper title normalization
feat(auto_fin): add multi-run cron schedules for intraday refinement
- Defined three auto_fin cron jobs at 09:30, 11:30, and 18:00 Shanghai time for gradual report updates
- Each intraday run adds evidence cumulatively instead of replacing prior output wholly
- Updated daily_cookbook.yaml to register new cron schedules and remove legacy 12:00 cron
refactor(auto_fin_data): clean ETF code handling and page limits
- Replaced hardcoded DEFAULT_ETF_CODES with required non-empty config value "etf_codes"
- Added constants for major news and fund page limits to control pagination
- Improved ETF name extraction logic to handle missing fields consistently
fix(auto_fin_merge): fix report retrieval and merging logic
- Added support for getting current intra-day report in addition to previous day's report
- Modified merge template to include prior and current report sections for better context
- Adjusted report path handling to consistently use Path objects
test(auto_fin): add coverage for returns calculation and report retrieval
- Added test for returns when event occurs before close on non-trading day, checking next session entry
- Added test for previous and current report retrieval feeding merge context with disk files
- Extended test asserts for auto_fin cron schedule changes in config
style(daily_paper): reorder and cleanup imports
- Reorganized imports in _common.py for clarity and added missing collections.abc.Iterator import
- Cleaned up commented and unused imports across daily_paper steps
* feat: add configurable upstream mirror proxy
* style: format auto-fin data step
* fix: align cookbook mirrors and contracts
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(backend): improve workspace support for web clients
* fix(config): preserve the default workspace directory
* chore(reme): bump version to 0.4.1.5
- Update __version__ from 0.4.1.4 to 0.4.1.5 in initialization file
* fix(chat): disable builtin tools in read-only mode
* fix(agent): make builtin tools opt-in
* fix(list): tolerate files removed during mtime sort
* fix(chat): expose complete read-only job set
On macOS, mimetypes.guess_type() may not recognize .md files, causing
stat to report application/octet-stream and breaking test assertions.
Explicitly map .md files to text/markdown so the behavior is stable
across platforms.
* chore(benchmark): isolate dataset/workspaces/results per benchmark
- Move shared benchmark/{datasets,memory_workspaces,results} into per-benchmark subdirs benchmark/<name>/{dataset,workspaces,results}
- Update beam/longmemeval config.yaml and run.py path defaults
- Relocate longmemeval download.py to benchmark/longmemeval/ (downloads into dataset/ subdir); inline dataset download docs into README
- Update .gitignore: benchmark/*/{dataset,workspaces,results}/
- Move result-{beam,longmemeval}.md to benchmark/results_md/ and drop result- prefix; update README links
- Fix stale path refs in llm_judge.py and logs/demo_search_format.py
* feat(benchmark): add read tool to agentic answer and update BEAM results
- Add 'read' to job_tools in BaseAgenticAnswerStep for file reading capability
- Document read tool usage in lme/agentic_answer.yaml system prompt
- Update result-beam.md with latest evaluation scores (OVERALL: 0.623/0.580)
* feat(auto_memory): add source line-number markers for note traceability
- Add _format_history hook in AutoMemoryStep with line-number annotation
- Override in BeamAutoMemoryStep to prefix each turn with [Ln] for citation
- Add session_file variable to prompt templates for source marker paths
- Simplify repeated extraction rules by referencing system prompt
- Enhance agentic_answer search strategy (multi-search, read tool hint)
- Add warning log on ReadStep failure
* feat(beam): enhance auto_memory with source markers and pilot ingest tooling
* refactor(beam): rename max_chunk_words to max_segment_words, drop one-off pilot scripts
* feat: add CompressorStep and search_v2 dual-mode session compression
- Add CompressorStep (reme/steps/evolve/compressor.py) for direct LLM
text compression with optional query-guided relevance filtering
- Extend search_v2_step to support query-aware and query-independent
session transcript compression via _compress injected kwargs
- Refactor _source_format.py: split into render_chunk_entries +
join_chunk_entries; session chunks now render line-aligned with
L<n>: prefixes for verbatim/compressed parity
- Add JOB_TOOLS and INJECTED_JOB_KWARGS to BaseAgenticAnswerStep for
per-subclass tool and parameter injection
- LmeAgenticAnswerStep injects _search._compress payload to enable
query-aware compression during benchmark evaluation
- Record compression ablation results in result-longmemeval.md
- Add unit tests for CompressorStep and search compression paths
* refactor(compress): relax session compression to lenient format-preserving strategy and update LME results
* refactor(benchmark): make session compression config-driven via compress_session flag
Move session-transcript compression from LME hard-coded injection to a
runtime context flag set by evaluation.compress_session in each
benchmark config. Compression is off by default for both BEAM and LME,
and BaseAgenticAnswerStep now conditionally injects the _search compress
payload only when the flag is truthy.
* feat(lme/auto_memory): add source attribution markers with line numbers
Add _format_history to annotate each turn with [Ln] line numbers and
expose {session_file} in prompts so the agent can emit bare wikilink-style
source markers like [[session/dialog/s1.jsonl#L1-L2,L5-L6]] at the end
of factual entries. Consolidate the per-prompt body/format rules into
references to the system prompt to avoid drift, and add frontmatter-
protection guidance for the edit tool.
* feat: improve agentic answer prompt and update beam 100K results
- Strengthen abstention rule: prohibit extrapolation from related but
non-direct evidence
- Add multi-angle search after preliminary answer to check for
conflicting/supplementary/updated information
- Add max-iteration fallback to 'Information not found'
- Update beam.md with 100K results (agentscope 2.0.4.post1, from scratch)
including per-type token consumption and memory construction stats
- config.yaml: 100K dataset, 20 workers for BEAM evaluation
- run.py: add memory construction token usage tracking (default agent)
- Overall: 0.635 → 0.654 (+0.019), contradiction_resolution: 0.338 → 0.478
(+0.140), abstention: 0.500 → 0.525 (+0.025)
* feat(read): add session-aware formatting for read tool and update BEAM eval
- Add truncate_session_output in _file_io.py to render jsonl session
lines as [speaker @ time] content before byte-budget truncation
- Add read_step_format_session flag to ReadStep, honoring injected
job kwargs (precedence) and YAML fallback
- Inject read_step_format_session=True into BaseAgenticAnswerStep
so agentic answer reads render session transcripts human-readably
- Refine BEAM agentic_answer prompt: continue multi-angle search
after preliminary answer, forbid fabrication/extrapolation
- Update BEAM config to 1M variant and add sequential 100K-eval /
1M-build shell script
- Refresh benchmark/results_md/beam.md with latest results
* chore(config): disable expand_links in beam and lme search_v2 configs
* refactor(beam): drop one-off sequential 100K-eval-then-1M-build script
* fix(benchmark): add compressor job to beam config and fix BEAM clone instructions
- Add compressor job and compressor as_llm component to reme/config/beam.yaml
(aligned with lme.yaml) so that compress_session: true works for BEAM
- Add graceful degradation guard in search_v2._compress_session_entries:
when the compressor job is missing from the active config, log a warning
and skip compression instead of raising 'Job compressor not found'.
Skipped when there is no app_context so unit tests mocking run_job still
drive compression behavior.
- Fix BEAM download instructions in README.md/README_ZH.md: add mkdir -p
before cd benchmark/beam/dataset (the directory is gitignored and absent
in a fresh clone)
* fix(steps): guard compressor exceptions and fix ReadStep boolean override
1. search_v2: catch per-entry exceptions from run_job('compressor') inside
compress() so asyncio.gather never propagates a compressor failure (e.g.
temporary LLM outage). The failing entry keeps its original body while
remaining entries are still compressed, preserving already-retrieved
search results.
2. read: replace 'context_value or yaml_value' with an existence check so
that a runtime-injected False can explicitly disable a YAML-true
read_step_format_session flag.
Add focused unit tests for both paths.
* fix(search_v2): use existence check for strict_date_filter boolean override
Replace 'context_value or yaml_value' with an existence-based check so
that a runtime-injected False can explicitly disable a YAML-true
strict_date_filter flag, consistent with the read_step_format_session fix.
* refactor(search): simplify strict_date_filter fallback to truthiness-or
* style(test): rename unused param to satisfy pylint W0613
* refactor(benchmark): isolate per-benchmark assets and simplify LME agentic prompt
- Move shared benchmark/README, README_ZH, kill.sh, and results_md/*.md into
per-benchmark subdirs (benchmark/beam/, benchmark/longmemeval/) so each
benchmark owns its own docs, scripts, and result snapshots.
- Simplify lme/agentic_answer.yaml system prompt: drop verbose memory-system
description, keep search strategy, draft tool, and answer rules concise.
* docs(benchmark): update LME README_ZH results to latest eval run
---------
Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
* chore(benchmark): isolate dataset/workspaces/results per benchmark
- Move shared benchmark/{datasets,memory_workspaces,results} into per-benchmark subdirs benchmark/<name>/{dataset,workspaces,results}
- Update beam/longmemeval config.yaml and run.py path defaults
- Relocate longmemeval download.py to benchmark/longmemeval/ (downloads into dataset/ subdir); inline dataset download docs into README
- Update .gitignore: benchmark/*/{dataset,workspaces,results}/
- Move result-{beam,longmemeval}.md to benchmark/results_md/ and drop result- prefix; update README links
- Fix stale path refs in llm_judge.py and logs/demo_search_format.py
* feat(benchmark): add read tool to agentic answer and update BEAM results
- Add 'read' to job_tools in BaseAgenticAnswerStep for file reading capability
- Document read tool usage in lme/agentic_answer.yaml system prompt
- Update result-beam.md with latest evaluation scores (OVERALL: 0.623/0.580)
* feat(auto_memory): add source line-number markers for note traceability
- Add _format_history hook in AutoMemoryStep with line-number annotation
- Override in BeamAutoMemoryStep to prefix each turn with [Ln] for citation
- Add session_file variable to prompt templates for source marker paths
- Simplify repeated extraction rules by referencing system prompt
- Enhance agentic_answer search strategy (multi-search, read tool hint)
- Add warning log on ReadStep failure
* feat(beam): enhance auto_memory with source markers and pilot ingest tooling
* refactor(beam): rename max_chunk_words to max_segment_words, drop one-off pilot scripts
* feat: add CompressorStep and search_v2 dual-mode session compression
- Add CompressorStep (reme/steps/evolve/compressor.py) for direct LLM
text compression with optional query-guided relevance filtering
- Extend search_v2_step to support query-aware and query-independent
session transcript compression via _compress injected kwargs
- Refactor _source_format.py: split into render_chunk_entries +
join_chunk_entries; session chunks now render line-aligned with
L<n>: prefixes for verbatim/compressed parity
- Add JOB_TOOLS and INJECTED_JOB_KWARGS to BaseAgenticAnswerStep for
per-subclass tool and parameter injection
- LmeAgenticAnswerStep injects _search._compress payload to enable
query-aware compression during benchmark evaluation
- Record compression ablation results in result-longmemeval.md
- Add unit tests for CompressorStep and search compression paths
* refactor(compress): relax session compression to lenient format-preserving strategy and update LME results
* refactor(benchmark): make session compression config-driven via compress_session flag
Move session-transcript compression from LME hard-coded injection to a
runtime context flag set by evaluation.compress_session in each
benchmark config. Compression is off by default for both BEAM and LME,
and BaseAgenticAnswerStep now conditionally injects the _search compress
payload only when the flag is truthy.
* feat(lme/auto_memory): add source attribution markers with line numbers
Add _format_history to annotate each turn with [Ln] line numbers and
expose {session_file} in prompts so the agent can emit bare wikilink-style
source markers like [[session/dialog/s1.jsonl#L1-L2,L5-L6]] at the end
of factual entries. Consolidate the per-prompt body/format rules into
references to the system prompt to avoid drift, and add frontmatter-
protection guidance for the edit tool.
* feat: improve agentic answer prompt and update beam 100K results
- Strengthen abstention rule: prohibit extrapolation from related but
non-direct evidence
- Add multi-angle search after preliminary answer to check for
conflicting/supplementary/updated information
- Add max-iteration fallback to 'Information not found'
- Update beam.md with 100K results (agentscope 2.0.4.post1, from scratch)
including per-type token consumption and memory construction stats
- config.yaml: 100K dataset, 20 workers for BEAM evaluation
- run.py: add memory construction token usage tracking (default agent)
- Overall: 0.635 → 0.654 (+0.019), contradiction_resolution: 0.338 → 0.478
(+0.140), abstention: 0.500 → 0.525 (+0.025)
* feat(read): add session-aware formatting for read tool and update BEAM eval
- Add truncate_session_output in _file_io.py to render jsonl session
lines as [speaker @ time] content before byte-budget truncation
- Add read_step_format_session flag to ReadStep, honoring injected
job kwargs (precedence) and YAML fallback
- Inject read_step_format_session=True into BaseAgenticAnswerStep
so agentic answer reads render session transcripts human-readably
- Refine BEAM agentic_answer prompt: continue multi-angle search
after preliminary answer, forbid fabrication/extrapolation
- Update BEAM config to 1M variant and add sequential 100K-eval /
1M-build shell script
- Refresh benchmark/results_md/beam.md with latest results
* chore(config): disable expand_links in beam and lme search_v2 configs
* refactor(beam): drop one-off sequential 100K-eval-then-1M-build script
* fix(benchmark): add compressor job to beam config and fix BEAM clone instructions
- Add compressor job and compressor as_llm component to reme/config/beam.yaml
(aligned with lme.yaml) so that compress_session: true works for BEAM
- Add graceful degradation guard in search_v2._compress_session_entries:
when the compressor job is missing from the active config, log a warning
and skip compression instead of raising 'Job compressor not found'.
Skipped when there is no app_context so unit tests mocking run_job still
drive compression behavior.
- Fix BEAM download instructions in README.md/README_ZH.md: add mkdir -p
before cd benchmark/beam/dataset (the directory is gitignored and absent
in a fresh clone)
* fix(steps): guard compressor exceptions and fix ReadStep boolean override
1. search_v2: catch per-entry exceptions from run_job('compressor') inside
compress() so asyncio.gather never propagates a compressor failure (e.g.
temporary LLM outage). The failing entry keeps its original body while
remaining entries are still compressed, preserving already-retrieved
search results.
2. read: replace 'context_value or yaml_value' with an existence check so
that a runtime-injected False can explicitly disable a YAML-true
read_step_format_session flag.
Add focused unit tests for both paths.
* fix(search_v2): use existence check for strict_date_filter boolean override
Replace 'context_value or yaml_value' with an existence-based check so
that a runtime-injected False can explicitly disable a YAML-true
strict_date_filter flag, consistent with the read_step_format_session fix.
* refactor(search): simplify strict_date_filter fallback to truthiness-or
* style(test): rename unused param to satisfy pylint W0613
---------
Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
* feat: simplify local links and support line anchors
* fix: align line anchor tests with CI lint
* fix: preserve local links across file moves
* fix: encode markdown paths when rewriting links
* refactor(read): keep explicit line range parameters
* fix: simplify legacy link predicate compatibility
* docs: align local link behavior with implementation
* fix: skip unsupported markdown destination escapes
* fix: normalize workspace link paths across platforms
* fix: bound markdown link scanning
* fix: keep local link processing linear
* docs: clarify permissive markdown link parsing
* fix: handle local link processing failures
* refactor: limit file links to wikilink syntax
* docs: align wikilink contract with implementation
* fix: normalize dream and neighbor paths on Windows
* fix: resolve workspace path for neighbor expansion
- Introduce tolerant AutoFinAgentModel base class allowing extra fields in raw Agent outputs
- Replace strict models with tolerant ones for ETF, historical event, market selection, and report outputs
- Remove redundant field validators and allow empty defaults for key string fields
- Enhance historical source path resolution to safely filter invalid or out-of-workspace paths
- Add normalization of whitespace and validation to historical event references before processing
- Implement normalization in Topic and Market Agent selections to eliminate duplicates, blanks, unknowns
- Limit Topic Agent output to top 20 ETFs and ensure sorting and deduplication of events
- Normalize final Markdown report by removing redundant headers and providing safe fallbacks
- Update agent prompts to clarify task constraints and improve instruction consistency
- Add extensive tests for normalization, filtering, and safe source resolution for historical events
* feat(file_store): upgrade FAISS to HNSW index with async reindex and path constraint
- Replace IndexFlatIP with IndexHNSWFlat for better recall/speed tradeoff
- Add dynamic efSearch (limit * 5) scaled to query request size
- Add async_reindex option: background rebuild with generation-based invalidation
- Extract _delete_nodes() in LocalFileStore for subclass reuse
- Add unit tests for file store consistency
* fix: resolve pylint warnings in faiss store and test file
* refactor(file_store): replace generation-based reindex with event-flag worker
- Replace _reindex_generation/lock/task with a single long-lived worker
coroutine consuming an asyncio.Event flag; repeated submissions coalesce
- Use local index reference in vector_search to avoid TOCTOU on self._faiss_index
- Pass index explicitly to _set_ef_search for consistency
- Track _index_writes to re-arm reindex after concurrent writes
- Update tests to match new internal API
* fix: resolve pylint too-many-return-statements and implicit-booleaness warnings
* feat(file_store): add refine maintenance hook and incremental embedding backfill
- Add refine() idle-time maintenance hook to BaseFileStore/LocalFileStore
- FaissLocalFileStore: incremental vector add on backfill instead of full rebuild
- Dynamic tombstone compaction threshold scaled by index size
- Add RefineStoreStep with daily cron job (refine_store_cron)
- Enable faiss backend and embedding_store by default in default.yaml
- Add unit tests for faiss index maintenance
* chore(deps): promote faiss-cpu to core dependencies
faiss backend is now the default file_store, so faiss-cpu moves from
the optional [core] extra to the base dependencies list.
* feat: rename refine_store to optimize_index and add vecdb_path_constraint
- Rename refine_store step to optimize_index with cron job scheduling
- Add vecdb_path_constraint to file_store components
- Update default.yaml with optimize_index_cron and faiss backend comment
- Update memory_search docs (en/zh) for FAISS vector management
- Update unit tests for index maintenance
* feat(faiss): add embedding digest to reject stale sidecar after partial dump
Add _chunks_embedding_digest() that computes an order-independent SHA-256
over (chunk_id, float16 embedding) pairs. The digest is written into the
idmap sidecar at dump time and verified at load time. A mismatch means the
sidecar vectors belong to a different chunk generation than the authoritative
JSONL — detectable even when the live-ID set is unchanged (same-ID in-place
update crash window).
Add test_faiss_rejects_stale_sidecar_after_partial_dump reproducing the
crash-between-writes scenario and asserting digest-based rejection.
Compress verbose docstrings/comments in existing tests for pylint line
budget.
---------
Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
- Add AutoFinHistoricalDirectionReference model to classify historical events by direction
- Remove AutoFinHistoricalSimilarity and related similarity score usage
- Update AutoFinMarketSelection to handle same and opposite direction event lists
- Adjust AutoFinMarketStep to calculate forecasts based on equal weights and direction signs
- Change market.yaml instructions to require direction classification instead of similarity scoring
- Modify tests to reflect direction-based classification and verify uniqueness across direction groups
- Improve DingTalkWaitStep to support reconnect on server request with proper disconnect reason handling
content.split("\n") yields a trailing empty element for any file ending in a
newline, inflating total by 1 and letting a start_line one past real EOF be
silently accepted instead of rejected. Mirrors the trailing-newline correction
default_file_chunker already applies, and matches the large-file path's
line-by-line count.
Fixes#388
- Separate candidate source file resolution from event resolution logic
- Allow fallback to date-derived daily news file if original source is missing
- Check existence and validity of historical source files more robustly
- Handle multiple candidate source files and aggregate matches before validation
- Gather and log resolution limitations without stopping processing
- Return resolved events along with a list of resolution warnings
- Update related code to consume new return signature and merge limitations
- Add detailed validation on source path relativity and file naming conventions
* feat: add ssh proxy
* feat: add ssh proxy
* feat: add ssh proxy
* feat: add ssh proxy
* feat: add prompt
* feat: add agent wrapper
* feat: add agent wrapper
* feat: add agent wrapper
* feat: add tushare skill
* feat: add tushare skill
* feat: add tushare skill
* feat: add none stream
* chore(deps): update dependency versions in pyproject.toml
- Bump claude-agent-sdk from 0.2.123 to 0.2.126
- Upgrade pre-commit to version 4.6.1 or higher
- Upgrade pytest to version 9.1.1 or higher
* feat(agent_wrapper): add session compaction support and unify session commands
- Introduce compact_session method to BaseAgentWrapper and implement it in AsAgentWrapper, CcAgentWrapper, and CodexAgentWrapper
- Add session_command module with SessionCommandResult dataclass and handle_session_command function for /clear and /compact commands
- Update __init__.py exports to include session_command handlers
- Modify DingTalkWaitStep to handle session commands via handle_session_command function
- Remove streaming mode from DingTalkWaitStep and simplify reply handling to final Markdown replies only
- Add unit tests for session compaction methods and session command handling across wrappers and DingTalk integration
- Clean up and remove obsolete streaming and card rendering code from DingTalk wait step
- Adjust daily_cookbook.yaml to remove stream and card_update_interval config entries for DingTalk wait step
* feat(auto_fin): add Auto Fin simulated portfolio cookbook workflow
- Add comprehensive Auto Fin schema exports for multiple models and enums
- Implement base class and helpers for Auto Fin analysis steps
- Create file, state, and formatting utilities for Auto Fin with atomic file writes and locking
- Define Auto Fin pipeline with four analysis agents: backtest, event, portfolio, and US correlation
- Register Auto Fin package in cookbook workflows and schema initialization
- Add detailed documentation in markdown describing the system design, workflow, and data contracts
* feat(outbound_proxy): add application-scoped outbound HTTP proxy components
- Introduce BaseOutboundProxy and OutboundProxyEndpoint as core contracts
- Implement FixedHttpOutboundProxy for external HTTP proxy integration
- Add SshHttpOutboundProxy providing SSH-backed local HTTP proxy tunnels
- Register outbound proxy components in component registry and enumeration
- Update components package to include outbound_proxy module
- Add dependency on pproxy for SSH HTTP proxy bridging
- Include comprehensive unit tests covering proxy lifecycle, validation,
environment merging, error handling, readiness, and monitoring mechanisms
* refactor(network): replace SSH proxy with explicit HTTP outbound proxy
- Remove SSH proxy helper implementation and references in codebase
- Add support for explicit HTTP proxy URL in arXiv and HuggingFace clients
- Modify clients to use async context manager for consistent resource handling
- Update daily paper steps to forward outbound proxy configuration explicitly
- Change tests to cover new proxy usage model and remove SSH proxy mocks
- Add outbound proxy component configuration in daily_cookbook.yaml
- Ensure proxy URL usage disables environment trust in HTTP clients
- Fix app context component enum access to be defensive against missing keys
* feat(agent_wrapper): add managed proxy support for command environments
- Introduce BaseOutboundProxy binding in BaseAgentWrapper for outbound proxy management
- Add bash_environment and command_proxy_environment properties to apply proxy settings
- Update WorkspaceBackend instantiation in AsAgentWrapper to use bash_environment
- Inject managed proxy export commands into Claude Code Bash commands via hooks
- Enhance CodexAgentWrapper to include managed proxy in shell environment policy
- Modify daily_cookbook.yaml steps to specify outbound_proxy as default where needed
- Add comprehensive unit tests verifying managed proxy injection and environment isolation
- Ensure subprocess_environment remains unchanged while proxy is applied selectively to commands
* refactor(memory): replace search job_tools with memory in daily cookbook config
- Change workspace_dir default from .reme to reme_workspace
- Replace search job_tools with memory across multiple components and jobs
- Update descriptions to reflect long-term memory retrieval instead of search
- Modify system prompts to instruct using memory for retrieving notes
- Adjust unit tests to verify memory job_tools and job presence instead of search
- Ensure consistency in configuration and tests for memory backend usage
* refactor(config): rename memory to memory_search in daily cookbook config
- Change all occurrences of "memory" to "memory_search" in job_tools and job definitions
- Update related system prompts to reflect the new memory_search terminology
- Modify unit tests to assert the presence of memory_search instead of memory
- Ensure consistency across skills, job tools, and backend configurations in multiple components
* feat(auto_fin): add deterministic quantitative research and ranking fusion
- Introduce new schema models: EtfScore, RankingMetrics, ExtremeAnalysis,
DimensionRanking, and FusionRanking to represent deterministic research outputs
- Add ranking data to event, backtest, us_correlation, and portfolio analysis outputs
- Implement ranking_section renderer to format Top20 scores and diagnostics in Markdown
- Develop AutoFinQuantStep for deterministic ETF ranking using TuShare data, Polars,
and a custom extremely randomized tree ensemble
- Integrate quantitative rankings into backtest and portfolio analysis steps and reports
- Extend auto_fin pipeline with new quant_enabled and quant_required config options
- Enforce ranking constraints like unique codes, contiguous ranks, and normalized fusion weights
- Update analysis YAMLs with rules limiting data freshness, universe, and ranking usage
- Incorporate ranking outputs into all major markdown report bodies in Auto Fin pipeline
- Add concurrency-limited asynchronous TuShare client to fetch required market data
- Introduce cross-sectional rank correlation and NDCG metrics for ranking quality evaluation
* feat(auto_fin): implement stage-wise notification and reporting for analysis pipeline
- Refactor notification config in daily_cookbook.yaml to support dispatch steps
- Update AutoFinNotificationStep to deduplicate notifications per run stage
- Add _notify_stage method in pipeline to send notifications for each analysis stage
- Implement persistence and notification for event, backtest, US correlation, and portfolio stages
- Modify pipeline flow to persist reports and notify after each stage completion
- Adjust metadata to track notifications and errors per stage
- Update tests to verify stage-wise notification sending and deduplication
- Remove older combined report persistence in favor of modular stage handling
* feat(auto_fin): add outbound proxy support for Tushare API usage
- Introduce BaseOutboundProxy reference in AutoFinPipelineStep and AutoFinQuantStep
- Update TushareResearchClient and trade calendar fetch to accept and use proxy URL
- Create _ProxiedTushareApi adapter to route Tushare requests via explicit HTTP proxy
- Modify create_tushare_api utility to optionally return proxied API client
- Add unit tests covering proxy forwarding and client behavior with managed proxies
- Ensure proxy usage respects explicit proxy URL over environment fallback
- Integrate outbound proxy into data fetching and quantitative research steps
* feat(auto_fin): enforce checkpoint time validation and add state models
- Introduce AnalysisState base class and specific states for event, backtest, and US correlation analyses
- Replace analysis output types with corresponding state classes in run schemas
- Add require_checkpoint_reached method to validate decision_at/data_cutoff against current time
- Enforce checkpoint time checks before analysis steps in event, backtest, portfolio, and quant analyses
- Refactor quant data loading to include adjustment factors and apply price adjustments without fallback
- Update analysis YAML docs to require real-time checkpoint validation and forbid using future data
- Improve portfolio run serialization by excluding redundant legacy fields and nested proposed actions
- Add helper to extract readable sections from persisted checkpoint documents
- Fix event analysis output validation to reject events and sources with future timestamps
* feat(auto_fin): auto-select latest reached checkpoint if none specified
- Extend checkpoint config to accept empty string for auto selection
- Add static method to compute latest checkpoint reached by current time
- Modify pipeline step to auto-select checkpoint based on trade calendar and time
- Adjust force flag default depending on whether checkpoint is explicit or auto
- Log details when checkpoint is auto-selected to improve observability
- Add comprehensive tests for auto checkpoint selection logic and edge cases
- Remove deprecated default and required constraints from force parameter in config
* refactor(auto_fin): unify datetime comparison with compare_datetimes utility
- Replace direct datetime comparisons with compare_datetimes function calls
- Use cmp_to_key with compare_datetimes for sorting datetime tuples and lists
- Update validation logic in backtest, event, analysis, and ledger modules for consistent datetime handling
- Add unit tests to verify handling of naive and aware datetime comparisons in event and backtest validations
- Ensure marked_at and interval_end timestamps are set and compared consistently using compare_datetimes
- Improve correctness of ordering and conditional checks related to timestamps throughout auto_fin steps and ledger code
* feat(auto_fin): add datetime comparison helper for mixed timezone data
- Implement compare_datetimes function to handle naive and aware datetimes
- Ensure naive datetime is interpreted in the known timezone of the counterpart
- Facilitate comparisons between legacy and timezone-aware Auto Fin data
- Add module docstring explaining purpose of the helpers
* docs(auto_fin): enforce unique ETF representative per sub-theme in analysis rules
- Update backtest.yaml to recommend or highlight only one ETF per sub-theme for ETF analyses
- Modify event.yaml to map only one representative ETF per sub-theme, avoiding duplicate recommendations
- Revise portfolio.yaml to restrict holdings/buys to a single ETF per sub-theme, preventing repeated buys of highly overlapping ETFs
- Adjust us_correlation.yaml to retain only one representative A-share ETF per sub-theme for mapping or recommendation
- Add test to verify presence of new sub-theme uniqueness guidance in step prompts
* feat(auto_fin): separate draft model and include deterministic fusion ranking
- Introduce _PortfolioProposalDraft pydantic model for agent-authored fields before ranking
- Discard any "fusion_ranking" data from draft to prevent conflicts with canonical ranking
- Modify AutoFinPortfolioStep to receive draft, enrich with fusion_ranking, and produce final output
- Update tests to use _PortfolioProposalDraft and validate deterministic fusion ranking propagation
- Add async test verifying fusion ranking is correctly set in portfolio output with no errors
* refactor(auto_fin): rewrite and simplify Auto Fin schema and steps
- Remove legacy Auto Fin analysis step modules and helpers
- Replace complex ranking and portfolio models with simplified current-news models
- Update schema to focus on news-case workflow with new domain models
- Remove A-share decision checkpoints and backtest details from schema
- Simplify recommendation and decision output structures
- Clean up deprecated state and utility functions
- Update Auto Fin steps initialization to new pipeline steps only
- Improve uniqueness validation for themes and ETFs in research plan
* feat(auto_fin): implement full local cache and analysis workflow for Auto Fin
- Add AutoFinDataStep to prepare and cache daily TuShare data with lookback
- Add AutoFinAnalysisStep to analyze cached data and generate Markdown report
- Implement detailed time window, ETF filtering, and historical case validation
- Introduce YAML prompts for planning and decision-making steps
- Update .gitignore to include reme_workspace/
- Clean up config and import structure for auto_fin steps
- Remove old pipeline.py and consolidate functionality into new modules
- Use polars for efficient CSV reading and data processing
- Ensure atomic writes and strict JSON serialization for cache files
- Enforce rules on news timing, ETF universe, and historical case usage
* fix(auto_fin): restrict news data source to '财联社' in analysis and cache
- Update analysis templates to specify current news as from '财联社' only
- Modify news fetching functions to filter by source '财联社'
- Add validation method to check cached news source correctness
- Update news caching logic to exclude non-'财联社' news
- Enhance unit tests with multiple sources to ensure filtering works
- Confirm news API calls include source filter parameter as '财联社'
* refactor(auto_fin): convert I/O methods to asynchronous implementations
- Change _news, _dataset, and _theme_data methods to async for improved concurrency
- Move JSONL and CSV reading operations to asynchronous wrappers using asyncio.to_thread
- Remove synchronous _read_jsonl and _read_csv functions, integrate them as static async class methods
- Update cache validation methods to async, awaiting I/O operations accordingly
- Adjust usage of dataset and news retrieval in analysis step to await asynchronous methods
- Add async unit test to validate JSONL reading with unicode line separators
- Preserve existing functionality while enabling non-blocking file and data access
* fix(nx_file_graph): defer networkx import and improve dependency handling
- Move networkx import inside NxFileGraph constructor for lazy loading
- Raise ImportError with original exception context if networkx is missing
- Remove module-level fallback assignment of nx to None
- Expand test to block loading of multiple optional core dependencies eagerly
- Change exception type in test from ModuleNotFoundError to AssertionError
- Update test comments to reflect broader optional dependency checks
* feat(embedding_store): add quota retry delay mechanism for embedding requests
- Introduce quota_retry_delay parameter to configure wait time before retry on quota exhaustion
- Implement detection of insufficient quota errors in LocalEmbeddingStore without external SDK
- Add retry logic with custom delay when quota is insufficient during embedding requests
- Update configuration to set max_retries and quota_retry_delay defaults for embedding store
- Add unit tests covering quota exhaustion retry behavior with delay and opt-in control
- Ensure existing retry behavior remains unchanged if quota_retry_delay is not set
* feat(auto_fin): add detailed logging to analysis and data fetching steps
- Add _preview static method for bounded diagnostic output in analysis.py
- Log prompt start, completion, errors, and validation details in _reply method
- Add info logs for major processing steps in execute method of analysis.py
- Add debug and info logs for cache validation, data fetching, and pagination in data.py
- Log conditions for skipping reports and cache plans in data.py execute method
- Log download summaries and cache writes for news and ETF data
- Improve error logging with exception details in cache validation functions
- Ensure all logs include context such as record counts, paths, and parameters
* refactor(auto_fin): overhaul Auto Fin workflow and schema contracts
- Replace old Auto Fin schema models with comprehensive new data classes
- Remove legacy Auto Fin analysis step in favor of modular agent-based steps
- Introduce AutoFinAgentStep for validating structured agent replies
- Simplify data cleaning and JSONL writing utilities for news cache
- Remove synchronous and asynchronous dataset methods from analysis step
- Redefine Auto Fin analysis configuration for 360-day news retention and multi-step pipeline
- Remove embedded analysis prompt templates and replace with agent-driven logic
- Update __init__.py exports to match new step implementations and remove deprecated classes
- Improve error handling and validation in agent step reply processing
- Clean up redundant imports and unused code in analysis and data preparation modules
* feat(auto_fin): add detailed logging for analysis and data processing steps
- Add timing logs to measure agent prompt processing duration in analysis.py
- Log news cache hits and news write paths with record counts in data.py
- Include detailed info logs for news download start and completion in data.py
- Add start, progress, and completion logs with topic and event counts in history.py
- Log start and completion of merge step including path and ETF count in merge.py
- Add start and done logs with window and news counts in topic.py
* feat(auto_fin): enhance schema and steps with detailed ETF and event modeling
- Replace and add multiple AutoFin schema classes to support detailed ETF selection,
historical research, market analysis, forecast models, and report output with validation
- Implement Shanghai timezone normalization and strict validation in schema models
- Remove deprecated AutoFin analysis agent step and consolidate reply handling in base step
- Introduce AutoFinStep base class with shared helpers for prompt handling, data fetching,
logging, and JSONL file operations
- Add AutoFinDataStep to manage daily news data complete with schedule validation, caching,
and source validation logic
- Update cookbook configuration to customize auto_fin step parameters and simplify
outbound proxy settings
- Refactor imports and clean unused code for better maintainability
* feat(auto_fin): introduce detailed historical event resolution and market similarity analysis
- Add AutoFinHistoricalEventReference and AutoFinHistoricalSimilarity models for refined event referencing and similarity judgment
- Implement validation to ensure non-empty critical fields and uniqueness of historical news IDs
- Develop method to resolve Agent-selected historical event references from workspace files with strict path and existence checks
- Enrich historical events with market entry and future returns data after resolution
- Redesign market step to calculate similarity-weighted ETF forecasts based on matched historical event similarities
- Enforce validation on matched historical events for uniqueness and proper weight summation
- Simplify merge step output to final Markdown report without YAML frontmatter and redundant fields
- Update user instructions for history search, market, and merge steps to reflect new data structures and responsibilities
- Adjust test suite to cover new schema and step behavior changes, including enhanced validation and JSON output formats
* feat(auto_fin): add new cron jobs and output analysis jsonl
- Add new cron jobs auto_fin_1145_cron and auto_fin_1800_cron with auto_fin_steps
- Change auto_fin_0930_cron schedule to run Monday to Sunday
- Extend merge step to write analysis data to auto_fin_analysis.jsonl
- Update unit tests to verify new cron jobs and their steps configuration
* fix(auto_fin): improve atomic file write and refresh daily index
- Change temporary file naming to include UUID for uniqueness and hidden prefix
- Replace atomic write method from using Path.replace to os.replace with safe unlink
- Add import and use os.replace for safer file replace operation
- Refresh daily index after writing auto finance markdown and JSONL files
- Import and call refresh_day_index in merge step to update file index asynchronously
* docs(cookbook): add optional SSH proxy configuration in README files
- Introduce optional SSH proxy setup in auto-fin and daily_paper cookbooks
- Provide instructions to enable outbound proxy via `daily_cookbook.yaml` and environment variables
- Add `REME_PROXY_IP` and `REME_PROXY_ACCOUNT` environment variables descriptions in multiple README files
- Update English and Chinese README and README_ZH documents with proxy details
- Maintain consistent formatting of environment variable tables across documents
* fix(file_io): include schema_version in hidden metadata keys
- Added "schema_version" to _INDEX_HIDDEN_METADATA_KEYS in _daily_index.py
- Updated _render_notes_block to always include additional keys regardless of schema_version
fix(deps): move pproxy dependency to later in pyproject.toml
- Removed pproxy from early dependencies list
- Added pproxy back near the end of dependency list for better ordering
fix(outbound_proxy): require pproxy package for ssh_http proxy
- Added importlib.util check for pproxy package presence
- Raise RuntimeError if pproxy is not installed when using SSH HTTP outbound proxy
- Improved error message suggests installing reme-ai with 'core' extra
* docs(readme): update News section with new Cookbook workflows
- Clarify introduction of optional Cookbooks with Daily Paper and Auto Fin workflows
- Update English README to reflect both paper discovery and file-native ETF event research
- Revise Chinese README to include financial news and historical market data research capability
- Maintain announcement of paper acceptance at Findings of ACL 2026
* feat(auto_fin): add calculation results to final Markdown output
- Implement _calculation_results to summarize forecast for each ETF analyzed
- Include program-calculated results in the JSON input for the Markdown report
- Update YAML template to incorporate calculation results and adjust recommendation rules
- Refine recommendation logic to rely on event impact judgments combined with calculation outputs
- Modify tests to verify presence of calculation results and updated report content and format
* up prompt
* fix(keyword_index): ignore non-indexable chunks during keyword sync
- Add is_indexable method to base and BM25 keyword index classes to check text tokenizability
- Update local file store to exclude non-indexable chunks from expected document IDs to prevent rebuild
- Fix JSONL chunker to correctly handle Unicode line separator U+2028 inside JSON strings without splitting
- Add test to ensure non-empty but non-indexable chunk does not trigger keyword index rebuild
- Add test to verify U+2028 character does not cause incorrect JSONL record splitting
