* fix(embedding): isolate caches by vector space
* fix(embedding): stabilize cache space switching
* Revert "fix(embedding): stabilize cache space switching"
This reverts commit 74193c9a0a.
* fix(embedding): include resolved OpenAI endpoint in cache ID
* fix(embedding): stabilize cache space switching
* fix(embedding): isolate Ollama endpoint caches
* feat(daily-paper): add Hugging Face mirror switch
* refactor(daily-paper): simplify the HF mirror switch and warn on ignored env
The switch was a three-state bool|None where None preserved the legacy
environment-driven selection, but no production caller ever passes None --
collect.py always resolves an explicit bool. Collapse it to a plain bool
defaulting to False.
HF_MIRROR_URL no longer redirects traffic on its own, so warn when it is
configured while the mirror stays disabled; a mirror-only setup would
otherwise fall back to the official site with no signal. Both READMEs now
record the behavior change and stop presenting the two mirror variables as
symmetric -- arXiv remains environment-driven while Hugging Face is gated on
the job parameter.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(daily-paper): address mirror configuration feedback
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
- Remove retry/attempt model: QueryScheduler failures are now terminal
instead of requeued; QueryLease no longer carries an attempt number;
SandboxSpec.max_retries defaults to 0 and is deprecated
- Flatten workspace layout: drop the attempt-<n>/ directory level so
case results live directly under cases/<case_id>/
- Add fail-fast: new --fail-fast CLI flag, validation.fail_fast config
key, and ValidationFailFastError that cancels sibling workers and
writes a run-level failure.json on the first construction or query
error
- Clarify module boundaries: meta-reme/run.py prepares workspace and
runs initial validation; meta-reme/validation/run.py validates an
already prepared workspace
- Update design.md, validation README, and both benchmark configs
- Update tests for single-execution semantics and add fail-fast tests
* refactor: rebuild auto-fin and daily-paper cookbooks on structured-output agents
Rework the auto-fin and daily-paper cookbooks to run on structured-output
LLM agents instead of Claude Code agent wrappers, replace the SSH proxy with
data-source mirrors, and rewrite the affected unit tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(auto_fin): unify JSON output serialization and writing
- Extracted _write_output static method to serialize and write Pydantic models as compact JSON
- Replaced inline JSON dump and write calls with _write_output usage across auto_fin steps
- Added _report_path and _current_report for managing intra-day reports in AutoFinMergeStep
- Updated auto_fin merge step to write output via new _write_output method
- Enhanced news reading with caching in AutoFinHistoryStep
- Refined returns calculation to handle events before close on non-trading days correctly
feat(daily_paper): improve note path resolution and metadata handling
- Introduced iter_note_metadata generator for safe Markdown frontmatter iteration
- Added resolve_unique_note_path to avoid note filename conflicts on disk and in used titles
- Updated analyze, collect, digest, and select steps to use centralized constants and helpers
- Used utc_now_iso for consistent timestamping in metadata
- Replaced direct frontmatter loads with iter_note_metadata in collect and analyze steps
- Replaced hardcoded paper selection count with PAPER_COUNT constant in all relevant places
- Added _MAX_SELECT_ATTEMPTS constant in select step for attempt management
- Improved error messages for filename validation in daily paper title normalization
feat(auto_fin): add multi-run cron schedules for intraday refinement
- Defined three auto_fin cron jobs at 09:30, 11:30, and 18:00 Shanghai time for gradual report updates
- Each intraday run adds evidence cumulatively instead of replacing prior output wholly
- Updated daily_cookbook.yaml to register new cron schedules and remove legacy 12:00 cron
refactor(auto_fin_data): clean ETF code handling and page limits
- Replaced hardcoded DEFAULT_ETF_CODES with required non-empty config value "etf_codes"
- Added constants for major news and fund page limits to control pagination
- Improved ETF name extraction logic to handle missing fields consistently
fix(auto_fin_merge): fix report retrieval and merging logic
- Added support for getting current intra-day report in addition to previous day's report
- Modified merge template to include prior and current report sections for better context
- Adjusted report path handling to consistently use Path objects
test(auto_fin): add coverage for returns calculation and report retrieval
- Added test for returns when event occurs before close on non-trading day, checking next session entry
- Added test for previous and current report retrieval feeding merge context with disk files
- Extended test asserts for auto_fin cron schedule changes in config
style(daily_paper): reorder and cleanup imports
- Reorganized imports in _common.py for clarity and added missing collections.abc.Iterator import
- Cleaned up commented and unused imports across daily_paper steps
* feat: add configurable upstream mirror proxy
* style: format auto-fin data step
* fix: align cookbook mirrors and contracts
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(backend): improve workspace support for web clients
* fix(config): preserve the default workspace directory
* chore(reme): bump version to 0.4.1.5
- Update __version__ from 0.4.1.4 to 0.4.1.5 in initialization file
* fix(chat): disable builtin tools in read-only mode
* fix(agent): make builtin tools opt-in
* fix(list): tolerate files removed during mtime sort
* fix(chat): expose complete read-only job set
* docs: add Chinese ReMe blog article
* fix(docs): update wiki links and adjust SVG path coordinates
- Removed file extensions from wiki link texts for consistency
- Modified path coordinates for relation lines in SVG illustration
- Added an arrow path with fill color to indicate direction in SVG diagram
* docs(blog): expand ReMe user guide and invite community contributions
- Add detailed descriptions for different ReMe user groups including intelligent agents,
developers, researchers, engineers, and analysts
- Emphasize user control over data as editable Markdown files instead of black-box storage
- Introduce ReMe's long-term memory infrastructure accessible via multiple interfaces
- Highlight how ReMe can turn scattered information into personal knowledge networks
- Include a new "Welcome Contributions" section encouraging community involvement
- List areas for contribution such as integration, data sources, features, applications,
documentation, and issue feedback
On macOS, mimetypes.guess_type() may not recognize .md files, causing
stat to report application/octet-stream and breaking test assertions.
Explicitly map .md files to text/markdown so the behavior is stable
across platforms.
- Add meta-reme/git_manager.py for immutable baseline Git initialization
- Enhance prepare_workspace to build and init benchmark code bundle
- Refactor sandbox worker into build/queries modes with shared Application
lifecycle and per-query log isolation
- Add run_build, run_queries, export_evaluation to DockerReMeSandbox
- Add JobRequest and EvaluationQuery data models
- Support log_filepath parameter in get_logger for isolated query logs
- Add safety validation for case IDs and query artifact IDs
- Update tests for build/queries modes, log isolation, and workspace init
- Add sandbox README_ZH.md and update design docs
- Add meta-reme/run.py entry point loading a single --config YAML and
preparing the workspace via the data_preparation module.
- Add meta-reme/data_preparation/ (basic, lme, beam) normalizing BEAM and
LongMemEval training cases into the shared CaseSpec/SessionSpec/QuerySpec
contracts with reproducibility manifests and read-only case files.
- Add per-benchmark config_meta_reme.yaml and run_meta.py launchers for
beam and longmemeval that forward to the unified entry point.
- Add tests/unit_sandbox/test_meta_reme_run.py covering case selection,
read-only writes, empty-dir init, BEAM variant/time-anchor parsing, and
config loading.
- Update meta-reme/design.md to reflect the single-config CLI and the
renamed data_preparation/ structure.
Always apply conditional-line filtering so tagged prompt lines are removed unless the corresponding boolean flag is explicitly true. Add regressions for omitted flags with and without format variables.
Test: pytest tests/unit/test_prompt_handler.py -q
* chore(benchmark): isolate dataset/workspaces/results per benchmark
- Move shared benchmark/{datasets,memory_workspaces,results} into per-benchmark subdirs benchmark/<name>/{dataset,workspaces,results}
- Update beam/longmemeval config.yaml and run.py path defaults
- Relocate longmemeval download.py to benchmark/longmemeval/ (downloads into dataset/ subdir); inline dataset download docs into README
- Update .gitignore: benchmark/*/{dataset,workspaces,results}/
- Move result-{beam,longmemeval}.md to benchmark/results_md/ and drop result- prefix; update README links
- Fix stale path refs in llm_judge.py and logs/demo_search_format.py
* feat(benchmark): add read tool to agentic answer and update BEAM results
- Add 'read' to job_tools in BaseAgenticAnswerStep for file reading capability
- Document read tool usage in lme/agentic_answer.yaml system prompt
- Update result-beam.md with latest evaluation scores (OVERALL: 0.623/0.580)
* feat(auto_memory): add source line-number markers for note traceability
- Add _format_history hook in AutoMemoryStep with line-number annotation
- Override in BeamAutoMemoryStep to prefix each turn with [Ln] for citation
- Add session_file variable to prompt templates for source marker paths
- Simplify repeated extraction rules by referencing system prompt
- Enhance agentic_answer search strategy (multi-search, read tool hint)
- Add warning log on ReadStep failure
* feat(beam): enhance auto_memory with source markers and pilot ingest tooling
* refactor(beam): rename max_chunk_words to max_segment_words, drop one-off pilot scripts
* feat: add CompressorStep and search_v2 dual-mode session compression
- Add CompressorStep (reme/steps/evolve/compressor.py) for direct LLM
text compression with optional query-guided relevance filtering
- Extend search_v2_step to support query-aware and query-independent
session transcript compression via _compress injected kwargs
- Refactor _source_format.py: split into render_chunk_entries +
join_chunk_entries; session chunks now render line-aligned with
L<n>: prefixes for verbatim/compressed parity
- Add JOB_TOOLS and INJECTED_JOB_KWARGS to BaseAgenticAnswerStep for
per-subclass tool and parameter injection
- LmeAgenticAnswerStep injects _search._compress payload to enable
query-aware compression during benchmark evaluation
- Record compression ablation results in result-longmemeval.md
- Add unit tests for CompressorStep and search compression paths
* refactor(compress): relax session compression to lenient format-preserving strategy and update LME results
* refactor(benchmark): make session compression config-driven via compress_session flag
Move session-transcript compression from LME hard-coded injection to a
runtime context flag set by evaluation.compress_session in each
benchmark config. Compression is off by default for both BEAM and LME,
and BaseAgenticAnswerStep now conditionally injects the _search compress
payload only when the flag is truthy.
* feat(lme/auto_memory): add source attribution markers with line numbers
Add _format_history to annotate each turn with [Ln] line numbers and
expose {session_file} in prompts so the agent can emit bare wikilink-style
source markers like [[session/dialog/s1.jsonl#L1-L2,L5-L6]] at the end
of factual entries. Consolidate the per-prompt body/format rules into
references to the system prompt to avoid drift, and add frontmatter-
protection guidance for the edit tool.
* feat: improve agentic answer prompt and update beam 100K results
- Strengthen abstention rule: prohibit extrapolation from related but
non-direct evidence
- Add multi-angle search after preliminary answer to check for
conflicting/supplementary/updated information
- Add max-iteration fallback to 'Information not found'
- Update beam.md with 100K results (agentscope 2.0.4.post1, from scratch)
including per-type token consumption and memory construction stats
- config.yaml: 100K dataset, 20 workers for BEAM evaluation
- run.py: add memory construction token usage tracking (default agent)
- Overall: 0.635 → 0.654 (+0.019), contradiction_resolution: 0.338 → 0.478
(+0.140), abstention: 0.500 → 0.525 (+0.025)
* feat(read): add session-aware formatting for read tool and update BEAM eval
- Add truncate_session_output in _file_io.py to render jsonl session
lines as [speaker @ time] content before byte-budget truncation
- Add read_step_format_session flag to ReadStep, honoring injected
job kwargs (precedence) and YAML fallback
- Inject read_step_format_session=True into BaseAgenticAnswerStep
so agentic answer reads render session transcripts human-readably
- Refine BEAM agentic_answer prompt: continue multi-angle search
after preliminary answer, forbid fabrication/extrapolation
- Update BEAM config to 1M variant and add sequential 100K-eval /
1M-build shell script
- Refresh benchmark/results_md/beam.md with latest results
* chore(config): disable expand_links in beam and lme search_v2 configs
* refactor(beam): drop one-off sequential 100K-eval-then-1M-build script
* fix(benchmark): add compressor job to beam config and fix BEAM clone instructions
- Add compressor job and compressor as_llm component to reme/config/beam.yaml
(aligned with lme.yaml) so that compress_session: true works for BEAM
- Add graceful degradation guard in search_v2._compress_session_entries:
when the compressor job is missing from the active config, log a warning
and skip compression instead of raising 'Job compressor not found'.
Skipped when there is no app_context so unit tests mocking run_job still
drive compression behavior.
- Fix BEAM download instructions in README.md/README_ZH.md: add mkdir -p
before cd benchmark/beam/dataset (the directory is gitignored and absent
in a fresh clone)
* fix(steps): guard compressor exceptions and fix ReadStep boolean override
1. search_v2: catch per-entry exceptions from run_job('compressor') inside
compress() so asyncio.gather never propagates a compressor failure (e.g.
temporary LLM outage). The failing entry keeps its original body while
remaining entries are still compressed, preserving already-retrieved
search results.
2. read: replace 'context_value or yaml_value' with an existence check so
that a runtime-injected False can explicitly disable a YAML-true
read_step_format_session flag.
Add focused unit tests for both paths.
* fix(search_v2): use existence check for strict_date_filter boolean override
Replace 'context_value or yaml_value' with an existence-based check so
that a runtime-injected False can explicitly disable a YAML-true
strict_date_filter flag, consistent with the read_step_format_session fix.
* refactor(search): simplify strict_date_filter fallback to truthiness-or
* style(test): rename unused param to satisfy pylint W0613
* refactor(benchmark): isolate per-benchmark assets and simplify LME agentic prompt
- Move shared benchmark/README, README_ZH, kill.sh, and results_md/*.md into
per-benchmark subdirs (benchmark/beam/, benchmark/longmemeval/) so each
benchmark owns its own docs, scripts, and result snapshots.
- Simplify lme/agentic_answer.yaml system prompt: drop verbose memory-system
description, keep search strategy, draft tool, and answer rules concise.
* docs(benchmark): update LME README_ZH results to latest eval run
---------
Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
* feat(file_store): add ZvecLocalFileStore backend
- Implement ZvecLocalFileStore with native zvec collection for ANN search.
- Keep JSONL chunks as the source of truth; rebuild collection from chunks
when sidecar digest/dimension/HNSW M mismatch is detected.
- Add dedicated unit tests in tests/unit/test_zvec_file_store.py.
- Parametrize existing file_store consistency tests to cover both
LocalFileStore and ZvecLocalFileStore.
- Register the new backend in reme/components/file_store/__init__.py.
* fix(file_store): fix zvec collection sync and content validation, declare zvec dependency
* chore(benchmark): isolate dataset/workspaces/results per benchmark
- Move shared benchmark/{datasets,memory_workspaces,results} into per-benchmark subdirs benchmark/<name>/{dataset,workspaces,results}
- Update beam/longmemeval config.yaml and run.py path defaults
- Relocate longmemeval download.py to benchmark/longmemeval/ (downloads into dataset/ subdir); inline dataset download docs into README
- Update .gitignore: benchmark/*/{dataset,workspaces,results}/
- Move result-{beam,longmemeval}.md to benchmark/results_md/ and drop result- prefix; update README links
- Fix stale path refs in llm_judge.py and logs/demo_search_format.py
* feat(benchmark): add read tool to agentic answer and update BEAM results
- Add 'read' to job_tools in BaseAgenticAnswerStep for file reading capability
- Document read tool usage in lme/agentic_answer.yaml system prompt
- Update result-beam.md with latest evaluation scores (OVERALL: 0.623/0.580)
* feat(auto_memory): add source line-number markers for note traceability
- Add _format_history hook in AutoMemoryStep with line-number annotation
- Override in BeamAutoMemoryStep to prefix each turn with [Ln] for citation
- Add session_file variable to prompt templates for source marker paths
- Simplify repeated extraction rules by referencing system prompt
- Enhance agentic_answer search strategy (multi-search, read tool hint)
- Add warning log on ReadStep failure
* feat(beam): enhance auto_memory with source markers and pilot ingest tooling
* refactor(beam): rename max_chunk_words to max_segment_words, drop one-off pilot scripts
* feat: add CompressorStep and search_v2 dual-mode session compression
- Add CompressorStep (reme/steps/evolve/compressor.py) for direct LLM
text compression with optional query-guided relevance filtering
- Extend search_v2_step to support query-aware and query-independent
session transcript compression via _compress injected kwargs
- Refactor _source_format.py: split into render_chunk_entries +
join_chunk_entries; session chunks now render line-aligned with
L<n>: prefixes for verbatim/compressed parity
- Add JOB_TOOLS and INJECTED_JOB_KWARGS to BaseAgenticAnswerStep for
per-subclass tool and parameter injection
- LmeAgenticAnswerStep injects _search._compress payload to enable
query-aware compression during benchmark evaluation
- Record compression ablation results in result-longmemeval.md
- Add unit tests for CompressorStep and search compression paths
* refactor(compress): relax session compression to lenient format-preserving strategy and update LME results
* refactor(benchmark): make session compression config-driven via compress_session flag
Move session-transcript compression from LME hard-coded injection to a
runtime context flag set by evaluation.compress_session in each
benchmark config. Compression is off by default for both BEAM and LME,
and BaseAgenticAnswerStep now conditionally injects the _search compress
payload only when the flag is truthy.
* feat(lme/auto_memory): add source attribution markers with line numbers
Add _format_history to annotate each turn with [Ln] line numbers and
expose {session_file} in prompts so the agent can emit bare wikilink-style
source markers like [[session/dialog/s1.jsonl#L1-L2,L5-L6]] at the end
of factual entries. Consolidate the per-prompt body/format rules into
references to the system prompt to avoid drift, and add frontmatter-
protection guidance for the edit tool.
* feat: improve agentic answer prompt and update beam 100K results
- Strengthen abstention rule: prohibit extrapolation from related but
non-direct evidence
- Add multi-angle search after preliminary answer to check for
conflicting/supplementary/updated information
- Add max-iteration fallback to 'Information not found'
- Update beam.md with 100K results (agentscope 2.0.4.post1, from scratch)
including per-type token consumption and memory construction stats
- config.yaml: 100K dataset, 20 workers for BEAM evaluation
- run.py: add memory construction token usage tracking (default agent)
- Overall: 0.635 → 0.654 (+0.019), contradiction_resolution: 0.338 → 0.478
(+0.140), abstention: 0.500 → 0.525 (+0.025)
* feat(read): add session-aware formatting for read tool and update BEAM eval
- Add truncate_session_output in _file_io.py to render jsonl session
lines as [speaker @ time] content before byte-budget truncation
- Add read_step_format_session flag to ReadStep, honoring injected
job kwargs (precedence) and YAML fallback
- Inject read_step_format_session=True into BaseAgenticAnswerStep
so agentic answer reads render session transcripts human-readably
- Refine BEAM agentic_answer prompt: continue multi-angle search
after preliminary answer, forbid fabrication/extrapolation
- Update BEAM config to 1M variant and add sequential 100K-eval /
1M-build shell script
- Refresh benchmark/results_md/beam.md with latest results
* chore(config): disable expand_links in beam and lme search_v2 configs
* refactor(beam): drop one-off sequential 100K-eval-then-1M-build script
* fix(benchmark): add compressor job to beam config and fix BEAM clone instructions
- Add compressor job and compressor as_llm component to reme/config/beam.yaml
(aligned with lme.yaml) so that compress_session: true works for BEAM
- Add graceful degradation guard in search_v2._compress_session_entries:
when the compressor job is missing from the active config, log a warning
and skip compression instead of raising 'Job compressor not found'.
Skipped when there is no app_context so unit tests mocking run_job still
drive compression behavior.
- Fix BEAM download instructions in README.md/README_ZH.md: add mkdir -p
before cd benchmark/beam/dataset (the directory is gitignored and absent
in a fresh clone)
* fix(steps): guard compressor exceptions and fix ReadStep boolean override
1. search_v2: catch per-entry exceptions from run_job('compressor') inside
compress() so asyncio.gather never propagates a compressor failure (e.g.
temporary LLM outage). The failing entry keeps its original body while
remaining entries are still compressed, preserving already-retrieved
search results.
2. read: replace 'context_value or yaml_value' with an existence check so
that a runtime-injected False can explicitly disable a YAML-true
read_step_format_session flag.
Add focused unit tests for both paths.
* fix(search_v2): use existence check for strict_date_filter boolean override
Replace 'context_value or yaml_value' with an existence-based check so
that a runtime-injected False can explicitly disable a YAML-true
strict_date_filter flag, consistent with the read_step_format_session fix.
* refactor(search): simplify strict_date_filter fallback to truthiness-or
* style(test): rename unused param to satisfy pylint W0613
---------
Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
Make "Experience-driven enhancement method" point to the archived benchmark page while keeping the arXiv link.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(benchmark): archive ExpG tool-use results under toolmemory
Add ToolMemory benchmark materials and link them from the root and
benchmark READMEs so ReMe documents the ExpG tool-use enhancement work.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs: point ToolMemory news entry directly to the paper
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(benchmark): address ToolMemory review and pre-commit
Restore benchmark index READMEs, link ExpG to WangCan1178/ExpG instead
of ReMe version notes, and format tool_memory.py for CI hooks.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(benchmark): align ToolMemory client with official ReMe APIs
Drop ExpG-only request fields and non-official metadata handling so the archived client matches add/summary/retrieve Tool Memory endpoints.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(benchmark): fix trailing whitespace in ToolMemory READMEs
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: simplify local links and support line anchors
* fix: align line anchor tests with CI lint
* fix: preserve local links across file moves
* fix: encode markdown paths when rewriting links
* refactor(read): keep explicit line range parameters
* fix: simplify legacy link predicate compatibility
* docs: align local link behavior with implementation
* fix: skip unsupported markdown destination escapes
* fix: normalize workspace link paths across platforms
* fix: bound markdown link scanning
* fix: keep local link processing linear
* docs: clarify permissive markdown link parsing
* fix: handle local link processing failures
* refactor: limit file links to wikilink syntax
* docs: align wikilink contract with implementation
* fix: normalize dream and neighbor paths on Windows
* fix: resolve workspace path for neighbor expansion
Add the meta-reme subsystem that generates minimal, benchmark-specific
ReMe source bundles from an allowlist. The bundle_builder.py reads
build_bundle.yaml to produce three targets (default, lme, beam), each
validated by import/step-registration checks before being staged.
Also include the design document describing the search-driven harness
optimization loop and a sandbox integration test exercising the first
LongMemEval and BEAM 100K cases against generated bundles.
Add commit_memory_history() so the host controls when daily-memory
changes are checkpointed into the runtime workspace's local Git repo.
Session ingestion no longer commits implicitly; the host groups one or
many sessions into an explicit boundary with a chosen message.
- Initialize a Git repo in the runtime workspace on case setup
- Validate the configured daily_dir and stage only that path
- Allow empty commits so boundaries are always recorded
- Reset removes the old Git repo and reinitializes for the next case
- Export retains .git for offline history inspection
Enhance the Docker benchmark sandbox so one container can process
multiple cases sequentially and export only analysis-relevant artifacts:
- Add reset_case() to discard the active case while keeping the
installed candidate, venv, and worker for the next case.
- Add export(profile="analysis") (default) and export_full() to choose
between a selective analysis archive and the complete disposable case
tree; analysis export omits rebuildable indexes, caches, temp files,
and source resources.
- Introduce runtime-layout.json written by the worker with validated,
non-secret, case-relative paths so the host can build safe tar
excludes.
- Serialize job, export, and reset operations with a per-case lock.
- Pin a sandbox-runtime extra in pyproject.toml that omits cookbook-only
tushare, dingtalk-stream, pypdf, and polars, and install it in
Dockerfile.base.
- Set TMPDIR to a case-local directory so subprocess temp files stay
inside the disposable case root.
- Add unit tests for the new behaviors and a real-Docker LongMemEval
integration smoke test.
