* feat(bench): adding eval adapter for proactiveness on Pi-Bench
* Revise README for π-Bench evaluation suite
Updated the README to reflect the new project name and description.
* fix(bench): refining pi-bench scripts according to cr comments
* fix(bench): restore agent builtin tools in prebuilt toolkit
* chore(benchmark): isolate dataset/workspaces/results per benchmark
- Move shared benchmark/{datasets,memory_workspaces,results} into per-benchmark subdirs benchmark/<name>/{dataset,workspaces,results}
- Update beam/longmemeval config.yaml and run.py path defaults
- Relocate longmemeval download.py to benchmark/longmemeval/ (downloads into dataset/ subdir); inline dataset download docs into README
- Update .gitignore: benchmark/*/{dataset,workspaces,results}/
- Move result-{beam,longmemeval}.md to benchmark/results_md/ and drop result- prefix; update README links
- Fix stale path refs in llm_judge.py and logs/demo_search_format.py
* feat(benchmark): add read tool to agentic answer and update BEAM results
- Add 'read' to job_tools in BaseAgenticAnswerStep for file reading capability
- Document read tool usage in lme/agentic_answer.yaml system prompt
- Update result-beam.md with latest evaluation scores (OVERALL: 0.623/0.580)
* feat(auto_memory): add source line-number markers for note traceability
- Add _format_history hook in AutoMemoryStep with line-number annotation
- Override in BeamAutoMemoryStep to prefix each turn with [Ln] for citation
- Add session_file variable to prompt templates for source marker paths
- Simplify repeated extraction rules by referencing system prompt
- Enhance agentic_answer search strategy (multi-search, read tool hint)
- Add warning log on ReadStep failure
* feat(beam): enhance auto_memory with source markers and pilot ingest tooling
* refactor(beam): rename max_chunk_words to max_segment_words, drop one-off pilot scripts
* feat: add CompressorStep and search_v2 dual-mode session compression
- Add CompressorStep (reme/steps/evolve/compressor.py) for direct LLM
text compression with optional query-guided relevance filtering
- Extend search_v2_step to support query-aware and query-independent
session transcript compression via _compress injected kwargs
- Refactor _source_format.py: split into render_chunk_entries +
join_chunk_entries; session chunks now render line-aligned with
L<n>: prefixes for verbatim/compressed parity
- Add JOB_TOOLS and INJECTED_JOB_KWARGS to BaseAgenticAnswerStep for
per-subclass tool and parameter injection
- LmeAgenticAnswerStep injects _search._compress payload to enable
query-aware compression during benchmark evaluation
- Record compression ablation results in result-longmemeval.md
- Add unit tests for CompressorStep and search compression paths
* refactor(compress): relax session compression to lenient format-preserving strategy and update LME results
* refactor(benchmark): make session compression config-driven via compress_session flag
Move session-transcript compression from LME hard-coded injection to a
runtime context flag set by evaluation.compress_session in each
benchmark config. Compression is off by default for both BEAM and LME,
and BaseAgenticAnswerStep now conditionally injects the _search compress
payload only when the flag is truthy.
* feat(lme/auto_memory): add source attribution markers with line numbers
Add _format_history to annotate each turn with [Ln] line numbers and
expose {session_file} in prompts so the agent can emit bare wikilink-style
source markers like [[session/dialog/s1.jsonl#L1-L2,L5-L6]] at the end
of factual entries. Consolidate the per-prompt body/format rules into
references to the system prompt to avoid drift, and add frontmatter-
protection guidance for the edit tool.
* feat: improve agentic answer prompt and update beam 100K results
- Strengthen abstention rule: prohibit extrapolation from related but
non-direct evidence
- Add multi-angle search after preliminary answer to check for
conflicting/supplementary/updated information
- Add max-iteration fallback to 'Information not found'
- Update beam.md with 100K results (agentscope 2.0.4.post1, from scratch)
including per-type token consumption and memory construction stats
- config.yaml: 100K dataset, 20 workers for BEAM evaluation
- run.py: add memory construction token usage tracking (default agent)
- Overall: 0.635 → 0.654 (+0.019), contradiction_resolution: 0.338 → 0.478
(+0.140), abstention: 0.500 → 0.525 (+0.025)
* feat(read): add session-aware formatting for read tool and update BEAM eval
- Add truncate_session_output in _file_io.py to render jsonl session
lines as [speaker @ time] content before byte-budget truncation
- Add read_step_format_session flag to ReadStep, honoring injected
job kwargs (precedence) and YAML fallback
- Inject read_step_format_session=True into BaseAgenticAnswerStep
so agentic answer reads render session transcripts human-readably
- Refine BEAM agentic_answer prompt: continue multi-angle search
after preliminary answer, forbid fabrication/extrapolation
- Update BEAM config to 1M variant and add sequential 100K-eval /
1M-build shell script
- Refresh benchmark/results_md/beam.md with latest results
* chore(config): disable expand_links in beam and lme search_v2 configs
* refactor(beam): drop one-off sequential 100K-eval-then-1M-build script
* fix(benchmark): add compressor job to beam config and fix BEAM clone instructions
- Add compressor job and compressor as_llm component to reme/config/beam.yaml
(aligned with lme.yaml) so that compress_session: true works for BEAM
- Add graceful degradation guard in search_v2._compress_session_entries:
when the compressor job is missing from the active config, log a warning
and skip compression instead of raising 'Job compressor not found'.
Skipped when there is no app_context so unit tests mocking run_job still
drive compression behavior.
- Fix BEAM download instructions in README.md/README_ZH.md: add mkdir -p
before cd benchmark/beam/dataset (the directory is gitignored and absent
in a fresh clone)
* fix(steps): guard compressor exceptions and fix ReadStep boolean override
1. search_v2: catch per-entry exceptions from run_job('compressor') inside
compress() so asyncio.gather never propagates a compressor failure (e.g.
temporary LLM outage). The failing entry keeps its original body while
remaining entries are still compressed, preserving already-retrieved
search results.
2. read: replace 'context_value or yaml_value' with an existence check so
that a runtime-injected False can explicitly disable a YAML-true
read_step_format_session flag.
Add focused unit tests for both paths.
* fix(search_v2): use existence check for strict_date_filter boolean override
Replace 'context_value or yaml_value' with an existence-based check so
that a runtime-injected False can explicitly disable a YAML-true
strict_date_filter flag, consistent with the read_step_format_session fix.
* refactor(search): simplify strict_date_filter fallback to truthiness-or
* style(test): rename unused param to satisfy pylint W0613
* refactor(benchmark): isolate per-benchmark assets and simplify LME agentic prompt
- Move shared benchmark/README, README_ZH, kill.sh, and results_md/*.md into
per-benchmark subdirs (benchmark/beam/, benchmark/longmemeval/) so each
benchmark owns its own docs, scripts, and result snapshots.
- Simplify lme/agentic_answer.yaml system prompt: drop verbose memory-system
description, keep search strategy, draft tool, and answer rules concise.
* docs(benchmark): update LME README_ZH results to latest eval run
---------
Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
* chore(benchmark): isolate dataset/workspaces/results per benchmark
- Move shared benchmark/{datasets,memory_workspaces,results} into per-benchmark subdirs benchmark/<name>/{dataset,workspaces,results}
- Update beam/longmemeval config.yaml and run.py path defaults
- Relocate longmemeval download.py to benchmark/longmemeval/ (downloads into dataset/ subdir); inline dataset download docs into README
- Update .gitignore: benchmark/*/{dataset,workspaces,results}/
- Move result-{beam,longmemeval}.md to benchmark/results_md/ and drop result- prefix; update README links
- Fix stale path refs in llm_judge.py and logs/demo_search_format.py
* feat(benchmark): add read tool to agentic answer and update BEAM results
- Add 'read' to job_tools in BaseAgenticAnswerStep for file reading capability
- Document read tool usage in lme/agentic_answer.yaml system prompt
- Update result-beam.md with latest evaluation scores (OVERALL: 0.623/0.580)
* feat(auto_memory): add source line-number markers for note traceability
- Add _format_history hook in AutoMemoryStep with line-number annotation
- Override in BeamAutoMemoryStep to prefix each turn with [Ln] for citation
- Add session_file variable to prompt templates for source marker paths
- Simplify repeated extraction rules by referencing system prompt
- Enhance agentic_answer search strategy (multi-search, read tool hint)
- Add warning log on ReadStep failure
* feat(beam): enhance auto_memory with source markers and pilot ingest tooling
* refactor(beam): rename max_chunk_words to max_segment_words, drop one-off pilot scripts
* feat: add CompressorStep and search_v2 dual-mode session compression
- Add CompressorStep (reme/steps/evolve/compressor.py) for direct LLM
text compression with optional query-guided relevance filtering
- Extend search_v2_step to support query-aware and query-independent
session transcript compression via _compress injected kwargs
- Refactor _source_format.py: split into render_chunk_entries +
join_chunk_entries; session chunks now render line-aligned with
L<n>: prefixes for verbatim/compressed parity
- Add JOB_TOOLS and INJECTED_JOB_KWARGS to BaseAgenticAnswerStep for
per-subclass tool and parameter injection
- LmeAgenticAnswerStep injects _search._compress payload to enable
query-aware compression during benchmark evaluation
- Record compression ablation results in result-longmemeval.md
- Add unit tests for CompressorStep and search compression paths
* refactor(compress): relax session compression to lenient format-preserving strategy and update LME results
* refactor(benchmark): make session compression config-driven via compress_session flag
Move session-transcript compression from LME hard-coded injection to a
runtime context flag set by evaluation.compress_session in each
benchmark config. Compression is off by default for both BEAM and LME,
and BaseAgenticAnswerStep now conditionally injects the _search compress
payload only when the flag is truthy.
* feat(lme/auto_memory): add source attribution markers with line numbers
Add _format_history to annotate each turn with [Ln] line numbers and
expose {session_file} in prompts so the agent can emit bare wikilink-style
source markers like [[session/dialog/s1.jsonl#L1-L2,L5-L6]] at the end
of factual entries. Consolidate the per-prompt body/format rules into
references to the system prompt to avoid drift, and add frontmatter-
protection guidance for the edit tool.
* feat: improve agentic answer prompt and update beam 100K results
- Strengthen abstention rule: prohibit extrapolation from related but
non-direct evidence
- Add multi-angle search after preliminary answer to check for
conflicting/supplementary/updated information
- Add max-iteration fallback to 'Information not found'
- Update beam.md with 100K results (agentscope 2.0.4.post1, from scratch)
including per-type token consumption and memory construction stats
- config.yaml: 100K dataset, 20 workers for BEAM evaluation
- run.py: add memory construction token usage tracking (default agent)
- Overall: 0.635 → 0.654 (+0.019), contradiction_resolution: 0.338 → 0.478
(+0.140), abstention: 0.500 → 0.525 (+0.025)
* feat(read): add session-aware formatting for read tool and update BEAM eval
- Add truncate_session_output in _file_io.py to render jsonl session
lines as [speaker @ time] content before byte-budget truncation
- Add read_step_format_session flag to ReadStep, honoring injected
job kwargs (precedence) and YAML fallback
- Inject read_step_format_session=True into BaseAgenticAnswerStep
so agentic answer reads render session transcripts human-readably
- Refine BEAM agentic_answer prompt: continue multi-angle search
after preliminary answer, forbid fabrication/extrapolation
- Update BEAM config to 1M variant and add sequential 100K-eval /
1M-build shell script
- Refresh benchmark/results_md/beam.md with latest results
* chore(config): disable expand_links in beam and lme search_v2 configs
* refactor(beam): drop one-off sequential 100K-eval-then-1M-build script
* fix(benchmark): add compressor job to beam config and fix BEAM clone instructions
- Add compressor job and compressor as_llm component to reme/config/beam.yaml
(aligned with lme.yaml) so that compress_session: true works for BEAM
- Add graceful degradation guard in search_v2._compress_session_entries:
when the compressor job is missing from the active config, log a warning
and skip compression instead of raising 'Job compressor not found'.
Skipped when there is no app_context so unit tests mocking run_job still
drive compression behavior.
- Fix BEAM download instructions in README.md/README_ZH.md: add mkdir -p
before cd benchmark/beam/dataset (the directory is gitignored and absent
in a fresh clone)
* fix(steps): guard compressor exceptions and fix ReadStep boolean override
1. search_v2: catch per-entry exceptions from run_job('compressor') inside
compress() so asyncio.gather never propagates a compressor failure (e.g.
temporary LLM outage). The failing entry keeps its original body while
remaining entries are still compressed, preserving already-retrieved
search results.
2. read: replace 'context_value or yaml_value' with an existence check so
that a runtime-injected False can explicitly disable a YAML-true
read_step_format_session flag.
Add focused unit tests for both paths.
* fix(search_v2): use existence check for strict_date_filter boolean override
Replace 'context_value or yaml_value' with an existence-based check so
that a runtime-injected False can explicitly disable a YAML-true
strict_date_filter flag, consistent with the read_step_format_session fix.
* refactor(search): simplify strict_date_filter fallback to truthiness-or
* style(test): rename unused param to satisfy pylint W0613
---------
Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
* docs(benchmark): archive ExpG tool-use results under toolmemory
Add ToolMemory benchmark materials and link them from the root and
benchmark READMEs so ReMe documents the ExpG tool-use enhancement work.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs: point ToolMemory news entry directly to the paper
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(benchmark): address ToolMemory review and pre-commit
Restore benchmark index READMEs, link ExpG to WangCan1178/ExpG instead
of ReMe version notes, and format tool_memory.py for CI hooks.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(benchmark): align ToolMemory client with official ReMe APIs
Drop ExpG-only request fields and non-official metadata handling so the archived client matches add/summary/retrieve Tool Memory endpoints.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(benchmark): fix trailing whitespace in ToolMemory READMEs
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(search): add tool_context-scoped chunk dedup with TTL
Introduce _ToolContextDedupMixin shared by search/vector_search/bm25_search
to skip already-seen chunks within one agent tool_context. Per-context state
lives in app_context.metadata with configurable TTL (default 24h).
* feat(search): unify chunk answer rendering with merge and explicit empty messages
- Refactor SearchStep/VectorSearchStep/Bm25SearchStep to share format_chunks_answer for consistent source rendering and adjacent session-chunk merging.
- Distinguish empty results: ALL_RETURNED_MESSAGE when dedup removes everything vs NO_RESULTS_MESSAGE when nothing matched.
- Bump JsonlFileChunker default max_chars to 4000.
- Add unit tests for source-format merge and empty-result messages.
* refactor(config): reorganize file_chunker components and move jsonl max_chars into config
- Register explicit markdown/json/jsonl chunkers in beam.yaml and lme.yaml with markdown options (embed_toc, max_ast_sections, frontmatter handling) and jsonl max_chars=4000.
- Restrict default chunker to txt/log extensions.
- Revert JsonlFileChunker code default max_chars back to 2000; the 4000 value now lives in config.
* chore(benchmark): increase longmemeval num_items from 64 to 500
* refactor(search): split SearchStep into simplified and v2 variants, extract counter utility
- Extract global_counter_next from ApplicationContext into reme/utils/counter.py
as a standalone function operating on metadata dict with lazy initialization.
- Split SearchStep into two variants:
- SearchStep (simplified): inline chunk.id dedup, single-branch vector/keyword
optimization based on vector_weight, inline answer formatting.
- SearchV2Step (full): preserves _ToolContextDedupMixin with interval-subset-aware
dedup and format_chunks_answer with session-aware chunk merging.
- Update beam.yaml and lme.yaml to use search_v2_step for benchmark jobs.
- Rename existing search tests to test_search_v2_step_* and add new
test_search_step_* tests covering the simplified variant.
* fix: normalise missing trailing newline in _build_union_chunk to prevent line collision
* refactor: lazy-init counter tree in ApplicationContext metadata
- Remove hardcoded _counter_tree and _counter_tree_lock initialization
from ApplicationContext.metadata; rely on lazy initialization in
reme.utils.counter.global_counter_next on first call
- Set longmemeval num_items back to 500
- Remove obsolete trailing-newline collision tests
---------
Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
* feat(eval): add LongMemEval evaluation framework with tool_defaults date injection
- Add evaluation/longmemeval/ with run.py, config.yaml, and test scripts
- Add reme/config/longmemeval.yaml for evaluation-specific model config
- Add tool_defaults mechanism to as_agent_wrapper for injecting default
tool kwargs (uses setdefault so LLM-provided values take priority)
- Pass tool_defaults={'daily_write': {'date': day}} in auto_memory to
ensure notes always use the correct historical date
- Add timestamp interpolation (_interpolate_timestamps) in auto_memory
for filling missing created_at fields via linear interpolation
- Evaluation pipeline: ingest sessions -> dream -> search -> answer -> judge
- Uses qwen3.6-flash for memory, qwen3.7-max for answer/judge
* chore: gitignore logs/results/demo.py, keep empty dirs
* chore: update .gitignore
* feat(eval): add multiprocessing and session time filtering to longmemeval runner
- Replace async execution with synchronous + multiprocessing for parallel item evaluation - Add filter_future_sessions option to only ingest sessions <= question date - Add question_types filtering in config - Add result summary with binary accuracy and avg score - Update config defaults (oracle variant, 50 items, 32 workers) - Minor code style fixes in agent_wrapper and auto_memory
* feat: add bench_query_step with ReAct agent for benchmark query phase
- Add BenchQueryStep using agent_wrapper with search job tool
- Replace manual search+LLM answer in run.py with bench_query_job
- Remove unused answer LLM config from longmemeval.yaml
- Register benchmark step module in steps/__init__.py
* feat: add start_date/end_date time filter support for search job
- Add _extract_date_from_path to extract validated YYYY-MM-DD from chunk paths
- Add start_date/end_date filtering in _matches_search_filter
- Implement progressive recall in FaissLocalFileStore.vector_search
- Promote start_date/end_date from context to search_filter in SearchStep
- Add start_date/end_date parameters to search job in default.yaml
- Add unit tests for date filter functionality
* fix: validate/normalize date filters and harden _extract_date_from_path
Address three code-review comments on the time_filter search feature:
1. Validate/normalize start_date and end_date before string comparison.
_matches_search_filter does lexicographic comparison against path_date
(always canonical YYYY-MM-DD). Raw caller values like '2026-2-28' or
'abc' would produce silently wrong results. Now SearchStep normalizes
valid dates via extract_daily_date (with strptime fallback for
non-zero-padded input) and silently ignores invalid dates with a
logger.warning, removing them from the filter.
2. Clarify behavior for paths without embedded dates.
Added optional strict_date_filter parameter (default False). When True
and at least one date bound is active, chunks whose path yields no date
(e.g. digest/personal/topic.md) are excluded. When False (default),
the existing behavior is preserved — dateless paths pass through.
3. Harden _extract_date_from_path against non-standard suffixes.
Previously parts[1].split('.')[0] accepted '2026-05-18.anything' as a
valid date. Now only exact 'YYYY-MM-DD' (dir) and 'YYYY-MM-DD.md'
(day-index) forms are accepted.
* feat(eval): LLM-as-Judge per-type prompt routing, binary-only, progress tracking
- Remove 0-5 score metric, keep only binary (yes/no) classification
- Load per-question-type judge prompts from llm-as-judge.json
(temporal-reasoning, knowledge-update, single-session-preference, __default__)
- Replace SCORE_JUDGE_PROMPT with type-specific BINARY_JUDGE_PROMPT template
- judge_response(): parameter 'metric' -> 'question_type', returns single 'judgment'
- Summary output: add per-type accuracy breakdown, remove score stats
- Add progress tracking: background thread prints PROGRESS every 10min
- Add FINAL progress line and total elapsed time on completion
- Add --log-level, --reme-log-level, -q CLI arguments
- Parallel mode: pool.map -> pool.imap_unordered for real-time progress
- config.yaml: full oracle (10000 items), 32 workers, all question types
- Add kill.sh (process cleanup) and run_async.sh (background eval launcher)
* docs: add LongMemEval oracle evaluation results (61.6% accuracy)
* feat(bench): add MAX_ITERATION limit to BenchQueryStep and add _auto_memory.yaml
* feat: add golden session benchmark & eval_only mode with refined prompt
- Add benchmark/longmemeval/run_golden_session.py for golden session evaluation
- Refine PROMPTED_SYSTEM_PROMPT: concise answer rule, remove 'Information not found' fallback
- Add eval_only mode to run.py (--eval_only flag)
- Add multiple eval config variants (evalonly, full, test5)
- Add analyze_results.py for result parsing
- Update auto_memory.yaml, longmemeval.yaml, application_config
- Update result-longmemeval.md with latest evaluation results
- Add benchmark results to .gitignore
* update: refine answer prompts and increase max iteration to 6 - Tighten prompted-answer system prompt for more concise output - Comment out 'Information not found' fallback rule - Increase MAX_ITERATION from 5 to 6 in bench_query - Add recall_eval.py - Update evaluation results
* feat(chunker): add dedicated JSON and JSONL file chunkers (cherry-pick from upstream #325)
- Add JsonFileChunker: structure-aware chunking preserving nested key paths,
optional list-to-dict conversion, size measured by json.dumps() char count
- Add JsonlFileChunker: line-aligned sliding-window chunking with configurable
overlap, supports char/byte mode switching
- Register both chunkers in default.yaml (json for .json, jsonl for .jsonl)
- Add comprehensive unit tests (21 + 20 test cases)
* feat(service): add CLI service for local job execution (from upstream #334)
- Introduce CliService to execute single jobs locally without serving ports
- Add prepare_start_config and should_precheck_start functions for CLI job setup
- Update reme start command to use CLI service when job argument is provided
- Add show_metadata to client kwargs for optional CLI metadata output
- Add unit tests for CLI service functionality and configuration handling
* feat(steps): add BM25/vector search steps, Python execute step, and draft steps (from upstream #334)
- Add Bm25SearchStep for plain BM25 keyword search with tool_context deduplication
- Add VectorSearchStep for plain vector search with tool_context deduplication
- Add PythonExecuteStep to run Python code in subprocess with timeout handling
- Add AddDraftStep/ReadAllDraftStep for draft accumulation scoped by tool context
- Update SearchStep with tool_context dedup, dynamic default limit via REME_SEARCH_LIMIT env,
and candidate_multiplier default changed from 3.0 to 5.0
- Add comprehensive unit tests for all new steps
* feat(search): add tool context deduplication and improve search configuration (#321)
* feat(search): add tool context deduplication and improve search configuration
- Modify _make_tool methods to accept and inject tool_context_id parameter
- Add tool_context_id handling in AS and CC agent wrappers
- Increase search candidate multiplier from 3.0 to 5.0 in default config
- Extend HTTP client timeout from 30s to 3600s
- Add tool context deduplication logic to prevent duplicate search results
- Implement TTL-based expiration for seen chunks in tool contexts
- Add comprehensive unit tests for tool context deduplication behavior
- Update .gitignore to exclude longmemeval directory
- Add time import for timestamp functionality in search step
* refactor(search): replace time module with datetime for timestamp generation
- Removed unused time import
- Added static method _now_ts using datetime.timestamp
- Updated clock parameter to use _now_ts method instead of time.time
- Maintained same timestamp precision and functionality
* fix(file_io): fix risk of out-workspace paths (#322)
* fix(file_io): fix risk of out-workspace paths
* chore(file_io): remove unused unittest file
* fix(as_embedding): support both agentscope 2.0.2 and 2.0.3 (#323)
2.0.3 promoted `dimensions` to a required first-class constructor
argument while keeping a backfill from `parameters.dimensions`; 2.0.2
has no such argument and reads `dimensions` from `Parameters`. Keep
`dimensions` in `Parameters` for both versions and, when the model
constructor accepts `dimensions`, pass `dimensions=None` so 2.0.3's
backfill promotes it out of `parameters`.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Bump version to 0.4.0.7
* refactor: delegate LLM-as-Judge to answer_judge_step and update eval config/results
- run.py: replace inline judge logic with judge_response_via_job using app.run_job('answer_judge')
- longmemeval.yaml: expand benchmark configuration
- bench_query.py: enhance benchmark query step
- result-longmemeval.md: update evaluation results
- judge_all_plus_results.json: add judge all-plus results
* refactor: split longmemeval.yaml into lme.yaml/beam.yaml and unify job names
- Split reme/config/longmemeval.yaml into lme.yaml (LongMemEval) and beam.yaml (BEAM)
- Unify job names across both configs: agentic_answer, answer_judge, context_answer
- Update evaluation/longmemeval/run.py and evaluation/beam/run_beam_eval.py to use unified job names
- Update all evaluation config YAMLs to reference lme.yaml
- Add BEAM benchmark step implementations (agentic_answer, context_answer, llm_judge)
- Remove obsolete config_test5.yaml and test_5sessions.py
* eval: BEAM 100K & LongMemEval cleaned-S 评测结果记录
- BEAM 100K eval-only (32并发, 20 case): Agentic 0.631, Prompted 0.468
- LongMemEval final GT (500题): Agentic 89.0%, Prompted 83.6%
- 新增 benchmark/result-beam.md, benchmark/result-longmemeval.md
- benchmark/beam/config.yaml: num_workers=32
* refactor: restructure benchmark directory and clean up gitignore rules
- Consolidate benchmark outputs to benchmark/results/ with .gitkeep
- Remove old benchmark scripts, configs and result files from benchmark/beam/ and benchmark/longmemeval/
- Add datasets/README.md and datasets/README_EN.md with download instructions
- Add datasets/longmemeval/download.py and final_groundtruth_cleaned_s.json
- Add memory_workspaces .gitkeep placeholders
- Restructure .gitignore: fix duplicate entries, add BEAM dataset exclusion, refine logs/results ignore patterns
- Remove stale result-beam.md and result-longmemeval.md from project root
* chore: clean up longmemeval benchmark scripts and update dataset docs
- Remove obsolete longmemeval benchmark runner/stats scripts
- Update datasets/longmemeval README and add Chinese translation
- Clean up final_groundtruth_cleaned_s.json
* docs(benchmark): add reproduction guide for LongMemEval and BEAM
- Add bilingual README for benchmark runners (EN/ZH)
- Cover prerequisites, dataset download, run commands, configs, outputs, logs, and kill.sh
* refactor: migrate auto_memory steps from evolve to benchmark-specific modules
- Split auto_memory into beam and lme benchmark-specific implementations
- Add auto_memory.py and auto_memory.yaml under steps/benchmark/beam and steps/benchmark/lme
- Slim down evolve/auto_memory.py and auto_memory.yaml to shared base only
- Remove obsolete evolve/_auto_memory.yaml
- Update benchmark run.py, config YAMLs, and step __init__.py registrations
- Update llm_judge and context_answer minor adjustments
- Remove outdated test_lme_final_answer_review.py
* revert(as_agent_wrapper): sync with upstream/main
Remove local-only comment to keep file identical with upstream/main.
* style: add trailing commas in benchmark __init__.py __all__ lists
* chore: disable vector_weight range assertion in SearchStep
* chore: add tests/integration/logs/ to .gitignore
* refactor: replace scipy.stats.kendalltau with pure numpy implementation
scipy is not listed in project dependencies. Implement Kendall's tau-b
rank correlation using only numpy to remove the undeclared dependency.
* feat(benchmark): add binary score metrics, update BEAM 1M results, and improve LLM retry/prompt config
- benchmark/beam/run.py: add binary score calculation per rubric item and per-type/overall binary stats
- benchmark/beam/config.yaml: switch to 1M dataset, reduce workers to 18
- benchmark/result-beam.md: add 1M evaluation results with binary scores
- benchmark/result-longmemeval.md: minor formatting
- reme/config/beam.yaml: increase max_retries to 5 and add retry_delay 5.0 for all LLM components
- reme/config/lme.yaml: increase max_retries to 5 and add retry_delay for judge/prompted/bench components
- reme/steps/benchmark/lme/agentic_answer.yaml: improve search strategy and answer rules prompts
* fix(benchmark): fix line-too-long and add pylint disable for main()
* refactor(longmemeval): use single cleaned-S dataset with embedded ground truth
- Switch to agentscope-ai/ReMe_longmemeval_clean_s_v2 HuggingFace source
- Remove separate final_groundtruth_cleaned_s.json (ground truth now in data file)
- Simplify download.py to fetch only longmemeval_s_reme_cleaned.json
- Remove dataset.variant and dataset.ground_truth_path config options
- Update benchmark and datasets READMEs to reflect new workflow
- Update .gitignore for new dataset filename
* fix: rename loop variable to avoid pylint redefined-outer-name warning
* refactor(benchmark): restructure datasets/memory_workspaces into benchmark and simplify auto_memory steps
* refactor(benchmark): extract BaseAgenticAnswerStep into base module
- Add reme/steps/benchmark/base/agentic_answer.py with shared agentic answer logic
- Refactor beam/lme AgenticAnswerStep to inherit from BaseAgenticAnswerStep
- Simplify lme/context_answer.py and update context_answer.yaml
- Update result-longmemeval.md with latest evaluation results (agentic 91.0%)
* refactor(benchmark): remove context_answer steps and unused configs
- Remove beam/lme context_answer job definitions and step implementations
- Remove prompted LLM component from beam.yaml and lme.yaml
- Delete jinli_lme.yaml (no longer needed)
- Simplify benchmark run.py scripts
- Clean up .gitkeep files and update .gitignore
- Remove unused import in search.py
* chore: remove benchmark/results/.gitkeep
---------
Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
Co-authored-by: jinliyl <6469360+jinliyl@users.noreply.github.com>
Co-authored-by: imrewce <wce@pku.edu.cn>
Co-authored-by: Sen Huang <48879559+ployts@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(benchmark): add final answer review step for evaluation
- Introduce FinalAnswerReviewStep to handle answer validation
- Add final_answer_review.jsonl dataset with 24 evaluation cases
- Include detailed reasoning and golden check results for each case
- Support various question types including temporal reasoning and preferences
- Implement time consistency checks for session references
- Add comprehensive test coverage for different evaluation scenarios
* chore(benchmark): remove longmemeval final answer review file
- Removed final_answer_review.jsonl containing 23 evaluation records
- Deleted question_id mappings with detailed reasoning for golden answers
- Removed answer correctness assessments and session time validation checks
- Cleaned up benchmark dataset used for memory evaluation testing
- Eliminated JSONL format evaluation results for temporal reasoning tasks
- Removed references to various session IDs and time-based validations
* config(default): disable shell step configuration by commenting out
- Commented out the shell step configuration in default.yaml
- Disabled asynchronous shell command execution capability
- Removed shell step from available backend operations
- Preserved traverse backend configuration unchanged
* refactor(tests): remove unused shell job test from config parser tests
- Removed test_default_config_registers_shell_job function that was no longer needed
- Kept existing test for frontmatter chunk metadata configuration
- Cleaned up test suite by removing obsolete test case
* feat(benchmark): add golden answer validation and session review for LongMemEval
- Introduce GoldenCheckStep to validate LongMemEval golden answers using structured verdicts
- Add SessionReviewStep to extract query/answer-relevant evidence from all sessions
- Implement concurrent session processing with configurable concurrency limits
- Create check_golden job configuration with lme_review and lme_judge agent wrappers
- Add Qwen3.7-plus model configuration for enhanced processing capabilities
- Include python_execute tool integration for agent-based reasoning and date validation
- Generate comprehensive JSON output with session summaries and validation verdicts
- Add run_check_golden.py script for batch processing across all LongMemEval samples
- Configure proper logging initialization with console and file output options
- Update component registry and file I/O modules to support new benchmark features
* feat(scripts): add script to summarize LongMemEval check_golden verdicts
- Parse check_golden.json files across all LongMemEval samples
- Calculate accuracy metrics for golden answers and session IDs
- Provide breakdown by question type with percentage calculations
- Add command line options for listing bad samples and JSON output
- Include progress tracking showing completed vs pending samples
- Display confidence scores and date sanity checks statistics
* refactor(benchmark): move golden check scripts to longmemeval directory
- Moved run_check_golden.py from scripts/ to benchmark/longmemeval/
- Moved stats_check_golden.py from scripts/ to benchmark/longmemeval/
- Updated path resolution to use parents[2] instead of parent.parent
- Added new --list-run-failed option to stats script
- Added logging directory constant and functions for tracking launched samples
- Enhanced stats output with launched count and run failure information
- Improved error reporting with run failure details and log file paths
* feat(benchmark): add LongMemEval agentic answer workflow with session extraction
- Add LmeAgenticAnswerStep, LmeAutoMemoryStep, and LmeExtractSessionStep to __init__.py
- Create shared helper render_with_source for displaying search results with session_id
- Implement agentic_answer step with vector_search, bm25_search, and extract_session_by_id tools
- Add auto_memory step to convert each session into search-friendly daily notes
- Create extract_session step to retrieve and analyze raw session content by session_id
- Update jinli_lme.yaml with auto_memory, vector_search, bm25_search, and agentic_answer jobs
- Configure lme_memory, lme_extract, and lme_agentic_answer agent wrappers
- Enhance search steps with include_source option to show session_id metadata
- Add proper session_id tracking and collision handling in daily note generation
* feat(benchmark): add LongMemEval agentic answer evaluation pipeline
- Add session_id tracking to agentic_answer.py result metadata
- Introduce run_agentic_answer.py driver for complete pipeline execution
- Implement auto_memory, update_index, and agentic_answer job orchestration
- Add concurrent execution with configurable limits and staggering
- Create aggregation script for collecting tool-call trails and results
- Add stats_agentic_answer.py for comprehensive result analysis
- Implement resume capability with existing output detection
- Generate aggregate.json with per-sample breakdown and tool call summaries
* feat(steps): add ClearPathsStep for cleaning workspace outputs before rebuild
- Introduce ClearPathsStep to remove stale workspace files/directories
- Add support for specifying paths and config_keys as targets to clear
- Implement safety checks to prevent deletion of files outside workspace
- Add logging for cleared paths and warnings for invalid paths
- Configure clear_paths_step in jinli_lme.yaml to clean daily_dir
- Add clear_paths_step to clean mem_answer.json before rebuilds
* feat(benchmark): add resume functionality to agentic answer runner
- Replace --force flag with --resume flag for controlling job execution
- By default every job reruns with clean rebuild behavior using config clear steps
- Add --resume option to skip samples whose output already exists and continue interrupted batches
- Update documentation to reflect new default clean rebuild behavior
- Modify job skipping logic to honor resume flag instead of force flag
- Update dry-run output to show correct todo jobs based on resume status
- Change default example command to use --resume for continuing interrupted runs
* feat(benchmark): generate JSONL output for check golden records
- Add write_check_golden_list function to create JSONL file
- Write all readable check_golden records as JSONL format
- Include check_golden_list path in stats output
- Display generated JSONL file path in summary report
- Maintain UTF-8 encoding with non-ASCII character support
* refactor(benchmark): rename answer judge step and integrate LME LLM judge
- Rename AnswerJudgeStep to LmeLlmJudgeStep and update imports
- Add new llm_judge configuration in jinli_lme.yaml
- Update run_agentic_answer.py to include llm_judge in pipeline
- Modify LmeLlmJudgeStep to read from query.json and answer.json
- Write LLM judgement results back to mem_answer.json
- Add command line options for start/end sample range selection
- Update aggregate.json generation to include LLM judgement data
- Add resume capability for llm_judge job based on judgement presence
* refactor(benchmark): rename answer judge step and integrate LME LLM judge
- Rename AnswerJudgeStep to LmeLlmJudgeStep and update imports
- Add new llm_judge configuration in jinli_lme.yaml
- Update run_agentic_answer.py to include llm_judge in pipeline
- Modify LmeLlmJudgeStep to read from query.json and answer.json
- Write LLM judgement results back to mem_answer.json
- Add command line options for start/end sample range selection
- Update aggregate.json generation to include LLM judgement data
- Add resume capability for llm_judge job based on judgement presence
* feat(steps): add wait_for_paths_step to block until workspace files exist
- Introduce WaitForPathsStep class that polls for required workspace-relative paths
- Add step registration with 'wait_for_paths_step' backend identifier
- Implement path validation to ensure targets are within workspace boundaries
- Add polling mechanism with configurable intervals via poll_seconds parameter
- Include logging functionality with log_every_seconds parameter for status updates
- Add metadata tracking of waited paths and duration in response object
- Register step in index module and expose in public API
- Configure step in jinli_lme.yaml to wait for session_review.json before golden check
- Add script rename from run_check_golden.py to run_golden_check.py with enhanced options
* feat(benchmark): enhance longmemeval benchmarking with concurrency and progress tracking
- Add benchmark extra dependency group with portalocker requirement
- Introduce concurrent execution support for golden_check and session_review workflows
- Add progress reporting interval option with real-time status updates
- Implement global throttling mechanism for session review requests using file locks
- Enhance golden check validation with current schema verification
- Add active task tracking and graceful shutdown handling
- Rename check_golden scripts to golden_check for consistency
- Update statistics reporting with correct/incorrect terminology instead of reasonable
- Add stale format detection and compatibility handling for verdict fields
- Include both_correct rate calculation in accuracy metrics
- Add concurrency and staggering options for better resource management
* ci(workflow): add Windows smoke test workflow
- Create new workflow file .github/workflows/windows-smoke.yml
- Configure workflow to trigger on push and pull request events
- Set up Python environment with version 3.11
- Install package dependencies using pip
- Run version job as smoke test for CLI functionality
- Enable concurrency control to prevent duplicate runs
- Use matrix strategy for Python version testing
* feat(benchmark): add retry mechanism and health check for session review
- Added retry configuration options (retry_initial_seconds, retry_max_seconds, retry_max_attempts) to jinli_lme.yaml
- Implemented exponential backoff retry logic with configurable parameters in session_review step
- Added output_is_healthy function to verify session_review.json integrity and absence of failed reviews
- Updated resume functionality to skip only healthy outputs instead of all existing files
- Integrated JSON parsing and validation to check for failed reviews in output files
- Enhanced error handling and logging for retry attempts and recovery scenarios
* feat(benchmark): add LongMemEval session review statistics script
- Create stats_session_review.py to summarize session_review.json artifacts
- Add command line options for listing failed, missing, and run failed samples
- Implement JSON output mode for programmatic consumption
- Calculate and display health statistics including total samples, healthy outputs, failed sessions
- Provide detailed failure information with session IDs and error messages
- Generate re-run commands for samples with failed reviews
- Add percentage calculations for better statistical overview
- Include support for multiple output formats and detailed logging
* feat(benchmark): add LongMemEval output cleanup script and enhance golden check retry logic
- Added clean_sample_outputs.py script to remove generated LongMemEval files while preserving source inputs
- Implemented configurable retry mechanism in golden_check.py with exponential backoff strategy
- Added retry parameters (initial/max seconds and max attempts) to control failure recovery behavior
- Integrated asyncio support for asynchronous sleep during retry intervals
- Configured default retry settings in jinli_lme.yaml with 5s initial and 300s maximum intervals
- Preserved core files (query.json, answer.json, session/) while cleaning generated artifacts
* feat(benchmark): add AppleDouble file cleanup to sample output cleaner
- Remove AppleDouble files starting with '._' recursively including under session/
- Add is_under helper function to check if path is inside parent directory
- Track targets in set to avoid duplicate processing
- Include AppleDouble files in cleanup targets when not already covered by existing targets
- Maintain dry-run mode as default behavior with --apply flag for actual deletion
* refactor(benchmark): update LongMemEval sample output cleaning script
- Add time and Iterator imports for enhanced functionality
- Add --progress-every argument to control progress reporting frequency
- Replace is_under function with iter_sample_targets generator
- Implement detailed progress tracking with timing measurements
- Add sample-by-sample processing with elapsed time reporting
- Include AppleDouble file detection within session directory
- Update target counting and deletion statistics display
- Add conditional progress updates based on progress-every setting
- Improve dry-run mode with would-delete indication
* chore(benchmark): increase initial interval for session review step
- Changed START_INTERVAL_SECONDS from 1.0 to 3.0 seconds
- Adjusted timing parameters for better benchmark stability
* refactor(benchmark): implement coordinated retry mechanism for session reviews
- Add retry gate condition to coordinate concurrent review attempts
- Implement wait_for_healthy_start_slot to handle sequential retries
- Create mark_retrying and mark_recovered functions to track retry states
- Update reply_with_retry to accept index parameter for coordination
- Add has_prior_retry logic to prevent race conditions during recovery
- Ensure proper cleanup of retry state on success or failure
- Maintain backward compatibility while adding coordination features
* chore(benchmark): adjust session review start interval timeout
- Changed START_INTERVAL_SECONDS from 3.0 to 5.0 seconds
- Increased initial delay for session review benchmark step
- Updated timeout configuration for improved stability
* refactor(benchmark): update session review concurrency and throttling mechanism
- Replace global throttle with per-process concurrency control
- Add concurrency parameter with default value of 30 in config
- Add start_interval_seconds parameter with default value of 2 seconds
- Change default concurrency from 3 to 1 in command line interface
- Update documentation to reflect new throttling behavior
- Implement semaphore-based concurrency limiting for review tasks
- Modify retry mechanism to use local locking instead of global files
- Remove portalocker dependency for cross-process throttling
* refactor(config): update session review configuration and concurrency settings
- Removed deprecated retry configuration parameters from jinli_lme.yaml
- Increased MAX_CONCURRENCY from 30 to 60 in session_review.py
- Reduced START_INTERVAL_SECONDS from 2.0 to 1.0 in session_review.py
- Cleaned up redundant backend specifications in configuration file
- Simplified agent wrapper configurations by removing obsolete retry settings
* feat(benchmark): enhance LME auto memory step with advanced scheduling and error handling
- Add datetime parsing functionality for LongMemEval timestamps with regex pattern
- Implement configurable concurrency limits with MAX_CONCURRENCY of 60
- Introduce retry mechanism with exponential backoff for agent interactions
- Add session filtering based on date comparison with question_date validation
- Create rate limiting with start interval control between requests
- Implement sophisticated retry coordination using asyncio conditions
- Add comprehensive error tracking for failed and filtered session extracts
- Remove deprecated concurrency parameter from jinli_lme.yaml configuration
- Add structured output validation in session review step
- Include detailed metadata reporting with session statistics and errors
* fix(benchmark): adjust default concurrency for auto_memory job
- Changed default concurrency from 3 to 1 for auto_memory job to prevent API overload
- Updated help text to reflect new default value of 1 for concurrency parameter
- Modified documentation to clarify concurrency behavior varies by job type
* refactor(search): replace hardcoded candidate multiplier with constant
- Introduced _CANDIDATE_MULTIPLIER constant set to 10
- Replaced hardcoded factor of 5 with _CANDIDATE_MULTIPLIER in BM25 search
- Replaced hardcoded factor of 5 with _CANDIDATE_MULTIPLIER in vector search
- Updated test to verify both search steps use ten times limit for candidates
- Imported VectorSearchStep and Bm25SearchStep in test module
- Added comprehensive test case for candidate count calculation logic
* feat(lme): add data inspection error handling with fallback mechanism
- Implemented non-retryable data inspection error markers detection
- Added _is_data_inspection_error method to identify inspection failures
- Created fallback handling for data inspection errors in auto memory extraction
- Added fallback handling for data inspection errors in session review
- Extended failed extracts tracking with non-retryable and fallback flags
- Separated fallback extracts from regular failed extracts in reporting
- Enhanced error logging with specific data inspection failure messages
- Updated metrics to track fallback extractions and reviews separately
- Maintained existing retry logic for other exception types
* feat(benchmark): enhance session review statistics with fallback tracking
- Add support for identifying and listing non-retryable fallback reviews
- Introduce --list-fallback argument to display fallback review details
- Separate retryable failures from non-retryable fallbacks in reporting
- Track fallback samples and sessions separately from failed ones
- Update console output to show both retryable and non-retryable categories
- Include fallback details in JSON output with reasons and session info
- Modify failure counting logic to distinguish between retryable and fallback reviews
* feat(benchmark): add question_id tracking and enhanced fallback reporting
- Add question_id function to extract query.question_id from data
- Initialize question_id_by_id dictionary to store question IDs by index
- Store question_id for each sample during data processing
- Enhance fallback output to include question IDs and session information
- Format sample labels with question IDs when available
- Display session IDs associated with each fallback case
* feat(benchmark): add question_id support and improve bad sample reporting
- Add question_id_for function to extract question_id from multiple sources
- Add sample_label function to format samples as idx(question_id) when available
- Store question_id in data dictionary during processing
- Change bad_golden and bad_sessions to store full records instead of just indices
- Update list_bad output to show formatted labels with question_id information
- Improve error reporting with more detailed sample identification
* feat(benchmark): enhance golden check stats with structured output
- Add related_session_ids function to extract session IDs from verdict records
- Create grouped_records function to group records by question type
- Replace flat list output with JSON-formatted grouped records in list_bad option
- Replace flat list output with JSON-formatted grouped records in list_bad_sessions option
- Maintain Chinese labels while adding structured data presentation
- Improve readability of bad verdict record display with hierarchical grouping
* feat(benchmark): update data structure for question indexing
- Replace sample_label with _idx field for index tracking
- Add question_id field to store _question_id values
- Maintain backward compatibility with empty string defaults
- Preserve existing session_id functionality
- Update data mapping to include new fields in grouped results
* refactor(benchmark): streamline golden answer verification process
- Replace relevance filtering with comprehensive information extraction
- Remove is_relevant field and simplify session summary structure
- Change relevant_info to extracted_info for clarity
- Update golden check logic to work with full extractions instead of filtered summaries
- Simplify prompt instructions to focus on complete information extraction
- Remove redundant schema validation and structured output requirements
- Adjust statistics calculation to match new extraction approach
- Update metadata field names to reflect extraction rather than relevance checking
* feat(benchmark): add selective file deletion option to clean_sample_outputs
- Add --filename argument to delete only specific root-level files
- Modify iter_sample_targets function to accept optional filenames filter
- Implement validation for root-level filename constraints
- Update function calls to pass filenames parameter
- Add example usage for selective file deletion in documentation
* feat(benchmark): add error count metrics to golden check statistics
- Added golden_bad, session_bad, and both_bad calculation fields
- Updated console output format to include error counts per question type
- Modified table display to show both accuracy rates and error numbers
- Enhanced statistical summary with additional error breakdown metrics
* test(search): update search step tests with include_source parameter
- Added include_source=False parameter to VectorSearchStep initialization
- Added include_source=False parameter to Bm25SearchStep initialization
- Maintained existing RuntimeContext parameters for both search steps
- Updated test calls to match new constructor signature with include_source option
* feat(memory): add ContextChecker component for context size management
* refactor(memory): restructure file-based memory tools and update imports
* docs(readme): update documentation with detailed architecture and components
* docs(readme): update Chinese documentation with enhanced memory management diagrams
* refactor(cookbook): move cookbook files to test directory and clean up docs
* docs(readme): update link path for old version documentation
* docs(readme): update documentation with improved architecture diagrams and component details
* docs(readme): update documentation with improved clarity and structure
* refactor(docs): update in-memory memory documentation
* docs(readme): add experiment reproduction link to quickstart guide
