mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-08-28 05:25:04 +00:00
7b1da5a9ee
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.11 (push) Waiting to run
Tests ReMe / Unit Tests - py3.12 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
Windows Smoke / CLI smoke - py3.11 (push) Waiting to run
* feat(eval): add LongMemEval evaluation framework with tool_defaults date injection
- Add evaluation/longmemeval/ with run.py, config.yaml, and test scripts
- Add reme/config/longmemeval.yaml for evaluation-specific model config
- Add tool_defaults mechanism to as_agent_wrapper for injecting default
tool kwargs (uses setdefault so LLM-provided values take priority)
- Pass tool_defaults={'daily_write': {'date': day}} in auto_memory to
ensure notes always use the correct historical date
- Add timestamp interpolation (_interpolate_timestamps) in auto_memory
for filling missing created_at fields via linear interpolation
- Evaluation pipeline: ingest sessions -> dream -> search -> answer -> judge
- Uses qwen3.6-flash for memory, qwen3.7-max for answer/judge
* chore: gitignore logs/results/demo.py, keep empty dirs
* chore: update .gitignore
* feat(eval): add multiprocessing and session time filtering to longmemeval runner
- Replace async execution with synchronous + multiprocessing for parallel item evaluation - Add filter_future_sessions option to only ingest sessions <= question date - Add question_types filtering in config - Add result summary with binary accuracy and avg score - Update config defaults (oracle variant, 50 items, 32 workers) - Minor code style fixes in agent_wrapper and auto_memory
* feat: add bench_query_step with ReAct agent for benchmark query phase
- Add BenchQueryStep using agent_wrapper with search job tool
- Replace manual search+LLM answer in run.py with bench_query_job
- Remove unused answer LLM config from longmemeval.yaml
- Register benchmark step module in steps/__init__.py
* feat: add start_date/end_date time filter support for search job
- Add _extract_date_from_path to extract validated YYYY-MM-DD from chunk paths
- Add start_date/end_date filtering in _matches_search_filter
- Implement progressive recall in FaissLocalFileStore.vector_search
- Promote start_date/end_date from context to search_filter in SearchStep
- Add start_date/end_date parameters to search job in default.yaml
- Add unit tests for date filter functionality
* fix: validate/normalize date filters and harden _extract_date_from_path
Address three code-review comments on the time_filter search feature:
1. Validate/normalize start_date and end_date before string comparison.
_matches_search_filter does lexicographic comparison against path_date
(always canonical YYYY-MM-DD). Raw caller values like '2026-2-28' or
'abc' would produce silently wrong results. Now SearchStep normalizes
valid dates via extract_daily_date (with strptime fallback for
non-zero-padded input) and silently ignores invalid dates with a
logger.warning, removing them from the filter.
2. Clarify behavior for paths without embedded dates.
Added optional strict_date_filter parameter (default False). When True
and at least one date bound is active, chunks whose path yields no date
(e.g. digest/personal/topic.md) are excluded. When False (default),
the existing behavior is preserved — dateless paths pass through.
3. Harden _extract_date_from_path against non-standard suffixes.
Previously parts[1].split('.')[0] accepted '2026-05-18.anything' as a
valid date. Now only exact 'YYYY-MM-DD' (dir) and 'YYYY-MM-DD.md'
(day-index) forms are accepted.
* feat(eval): LLM-as-Judge per-type prompt routing, binary-only, progress tracking
- Remove 0-5 score metric, keep only binary (yes/no) classification
- Load per-question-type judge prompts from llm-as-judge.json
(temporal-reasoning, knowledge-update, single-session-preference, __default__)
- Replace SCORE_JUDGE_PROMPT with type-specific BINARY_JUDGE_PROMPT template
- judge_response(): parameter 'metric' -> 'question_type', returns single 'judgment'
- Summary output: add per-type accuracy breakdown, remove score stats
- Add progress tracking: background thread prints PROGRESS every 10min
- Add FINAL progress line and total elapsed time on completion
- Add --log-level, --reme-log-level, -q CLI arguments
- Parallel mode: pool.map -> pool.imap_unordered for real-time progress
- config.yaml: full oracle (10000 items), 32 workers, all question types
- Add kill.sh (process cleanup) and run_async.sh (background eval launcher)
* docs: add LongMemEval oracle evaluation results (61.6% accuracy)
* feat(bench): add MAX_ITERATION limit to BenchQueryStep and add _auto_memory.yaml
* feat: add golden session benchmark & eval_only mode with refined prompt
- Add benchmark/longmemeval/run_golden_session.py for golden session evaluation
- Refine PROMPTED_SYSTEM_PROMPT: concise answer rule, remove 'Information not found' fallback
- Add eval_only mode to run.py (--eval_only flag)
- Add multiple eval config variants (evalonly, full, test5)
- Add analyze_results.py for result parsing
- Update auto_memory.yaml, longmemeval.yaml, application_config
- Update result-longmemeval.md with latest evaluation results
- Add benchmark results to .gitignore
* update: refine answer prompts and increase max iteration to 6 - Tighten prompted-answer system prompt for more concise output - Comment out 'Information not found' fallback rule - Increase MAX_ITERATION from 5 to 6 in bench_query - Add recall_eval.py - Update evaluation results
* feat(chunker): add dedicated JSON and JSONL file chunkers (cherry-pick from upstream #325)
- Add JsonFileChunker: structure-aware chunking preserving nested key paths,
optional list-to-dict conversion, size measured by json.dumps() char count
- Add JsonlFileChunker: line-aligned sliding-window chunking with configurable
overlap, supports char/byte mode switching
- Register both chunkers in default.yaml (json for .json, jsonl for .jsonl)
- Add comprehensive unit tests (21 + 20 test cases)
* feat(service): add CLI service for local job execution (from upstream #334)
- Introduce CliService to execute single jobs locally without serving ports
- Add prepare_start_config and should_precheck_start functions for CLI job setup
- Update reme start command to use CLI service when job argument is provided
- Add show_metadata to client kwargs for optional CLI metadata output
- Add unit tests for CLI service functionality and configuration handling
* feat(steps): add BM25/vector search steps, Python execute step, and draft steps (from upstream #334)
- Add Bm25SearchStep for plain BM25 keyword search with tool_context deduplication
- Add VectorSearchStep for plain vector search with tool_context deduplication
- Add PythonExecuteStep to run Python code in subprocess with timeout handling
- Add AddDraftStep/ReadAllDraftStep for draft accumulation scoped by tool context
- Update SearchStep with tool_context dedup, dynamic default limit via REME_SEARCH_LIMIT env,
and candidate_multiplier default changed from 3.0 to 5.0
- Add comprehensive unit tests for all new steps
* feat(search): add tool context deduplication and improve search configuration (#321)
* feat(search): add tool context deduplication and improve search configuration
- Modify _make_tool methods to accept and inject tool_context_id parameter
- Add tool_context_id handling in AS and CC agent wrappers
- Increase search candidate multiplier from 3.0 to 5.0 in default config
- Extend HTTP client timeout from 30s to 3600s
- Add tool context deduplication logic to prevent duplicate search results
- Implement TTL-based expiration for seen chunks in tool contexts
- Add comprehensive unit tests for tool context deduplication behavior
- Update .gitignore to exclude longmemeval directory
- Add time import for timestamp functionality in search step
* refactor(search): replace time module with datetime for timestamp generation
- Removed unused time import
- Added static method _now_ts using datetime.timestamp
- Updated clock parameter to use _now_ts method instead of time.time
- Maintained same timestamp precision and functionality
* fix(file_io): fix risk of out-workspace paths (#322)
* fix(file_io): fix risk of out-workspace paths
* chore(file_io): remove unused unittest file
* fix(as_embedding): support both agentscope 2.0.2 and 2.0.3 (#323)
2.0.3 promoted `dimensions` to a required first-class constructor
argument while keeping a backfill from `parameters.dimensions`; 2.0.2
has no such argument and reads `dimensions` from `Parameters`. Keep
`dimensions` in `Parameters` for both versions and, when the model
constructor accepts `dimensions`, pass `dimensions=None` so 2.0.3's
backfill promotes it out of `parameters`.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Bump version to 0.4.0.7
* refactor: delegate LLM-as-Judge to answer_judge_step and update eval config/results
- run.py: replace inline judge logic with judge_response_via_job using app.run_job('answer_judge')
- longmemeval.yaml: expand benchmark configuration
- bench_query.py: enhance benchmark query step
- result-longmemeval.md: update evaluation results
- judge_all_plus_results.json: add judge all-plus results
* refactor: split longmemeval.yaml into lme.yaml/beam.yaml and unify job names
- Split reme/config/longmemeval.yaml into lme.yaml (LongMemEval) and beam.yaml (BEAM)
- Unify job names across both configs: agentic_answer, answer_judge, context_answer
- Update evaluation/longmemeval/run.py and evaluation/beam/run_beam_eval.py to use unified job names
- Update all evaluation config YAMLs to reference lme.yaml
- Add BEAM benchmark step implementations (agentic_answer, context_answer, llm_judge)
- Remove obsolete config_test5.yaml and test_5sessions.py
* eval: BEAM 100K & LongMemEval cleaned-S 评测结果记录
- BEAM 100K eval-only (32并发, 20 case): Agentic 0.631, Prompted 0.468
- LongMemEval final GT (500题): Agentic 89.0%, Prompted 83.6%
- 新增 benchmark/result-beam.md, benchmark/result-longmemeval.md
- benchmark/beam/config.yaml: num_workers=32
* refactor: restructure benchmark directory and clean up gitignore rules
- Consolidate benchmark outputs to benchmark/results/ with .gitkeep
- Remove old benchmark scripts, configs and result files from benchmark/beam/ and benchmark/longmemeval/
- Add datasets/README.md and datasets/README_EN.md with download instructions
- Add datasets/longmemeval/download.py and final_groundtruth_cleaned_s.json
- Add memory_workspaces .gitkeep placeholders
- Restructure .gitignore: fix duplicate entries, add BEAM dataset exclusion, refine logs/results ignore patterns
- Remove stale result-beam.md and result-longmemeval.md from project root
* chore: clean up longmemeval benchmark scripts and update dataset docs
- Remove obsolete longmemeval benchmark runner/stats scripts
- Update datasets/longmemeval README and add Chinese translation
- Clean up final_groundtruth_cleaned_s.json
* docs(benchmark): add reproduction guide for LongMemEval and BEAM
- Add bilingual README for benchmark runners (EN/ZH)
- Cover prerequisites, dataset download, run commands, configs, outputs, logs, and kill.sh
* refactor: migrate auto_memory steps from evolve to benchmark-specific modules
- Split auto_memory into beam and lme benchmark-specific implementations
- Add auto_memory.py and auto_memory.yaml under steps/benchmark/beam and steps/benchmark/lme
- Slim down evolve/auto_memory.py and auto_memory.yaml to shared base only
- Remove obsolete evolve/_auto_memory.yaml
- Update benchmark run.py, config YAMLs, and step __init__.py registrations
- Update llm_judge and context_answer minor adjustments
- Remove outdated test_lme_final_answer_review.py
* revert(as_agent_wrapper): sync with upstream/main
Remove local-only comment to keep file identical with upstream/main.
* style: add trailing commas in benchmark __init__.py __all__ lists
* chore: disable vector_weight range assertion in SearchStep
* chore: add tests/integration/logs/ to .gitignore
* refactor: replace scipy.stats.kendalltau with pure numpy implementation
scipy is not listed in project dependencies. Implement Kendall's tau-b
rank correlation using only numpy to remove the undeclared dependency.
* feat(benchmark): add binary score metrics, update BEAM 1M results, and improve LLM retry/prompt config
- benchmark/beam/run.py: add binary score calculation per rubric item and per-type/overall binary stats
- benchmark/beam/config.yaml: switch to 1M dataset, reduce workers to 18
- benchmark/result-beam.md: add 1M evaluation results with binary scores
- benchmark/result-longmemeval.md: minor formatting
- reme/config/beam.yaml: increase max_retries to 5 and add retry_delay 5.0 for all LLM components
- reme/config/lme.yaml: increase max_retries to 5 and add retry_delay for judge/prompted/bench components
- reme/steps/benchmark/lme/agentic_answer.yaml: improve search strategy and answer rules prompts
* fix(benchmark): fix line-too-long and add pylint disable for main()
* refactor(longmemeval): use single cleaned-S dataset with embedded ground truth
- Switch to agentscope-ai/ReMe_longmemeval_clean_s_v2 HuggingFace source
- Remove separate final_groundtruth_cleaned_s.json (ground truth now in data file)
- Simplify download.py to fetch only longmemeval_s_reme_cleaned.json
- Remove dataset.variant and dataset.ground_truth_path config options
- Update benchmark and datasets READMEs to reflect new workflow
- Update .gitignore for new dataset filename
* fix: rename loop variable to avoid pylint redefined-outer-name warning
* refactor(benchmark): restructure datasets/memory_workspaces into benchmark and simplify auto_memory steps
* refactor(benchmark): extract BaseAgenticAnswerStep into base module
- Add reme/steps/benchmark/base/agentic_answer.py with shared agentic answer logic
- Refactor beam/lme AgenticAnswerStep to inherit from BaseAgenticAnswerStep
- Simplify lme/context_answer.py and update context_answer.yaml
- Update result-longmemeval.md with latest evaluation results (agentic 91.0%)
* refactor(benchmark): remove context_answer steps and unused configs
- Remove beam/lme context_answer job definitions and step implementations
- Remove prompted LLM component from beam.yaml and lme.yaml
- Delete jinli_lme.yaml (no longer needed)
- Simplify benchmark run.py scripts
- Clean up .gitkeep files and update .gitignore
- Remove unused import in search.py
* chore: remove benchmark/results/.gitkeep
---------
Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
Co-authored-by: jinliyl <6469360+jinliyl@users.noreply.github.com>
Co-authored-by: imrewce <wce@pku.edu.cn>
Co-authored-by: Sen Huang <48879559+ployts@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| longmemeval | ||
| README.md | ||
| README_EN.md | ||
Dataset Download Guide
This directory contains datasets required for ReMe evaluation. Some datasets are large and excluded from Git version control — they must be downloaded manually.
LongMemEval (cleaned-S)
ReMe uses only the cleaned-S split of LongMemEval, hosted on HuggingFace: agentscope-ai/ReMe_longmemeval_clean_s_v2 (the script downloads via the hf-mirror.com mirror).
Download it with:
cd benchmark/datasets/longmemeval
# Download the cleaned-S data file (skipped automatically if it already exists)
python download.py
After downloading, the directory should contain:
| File | Description |
|---|---|
longmemeval_s_reme_cleaned.json |
cleaned-S dataset with ground truth fields included |
download.py |
Download script (included in repo) |
Note
: The download script uses hf-mirror.com by default. To use a different mirror, modify
BASE_URLindownload.py.
Once the download completes, follow benchmark/README.md to run the LongMemEval evaluation.
BEAM
BEAM is a public repository. Clone it directly into the benchmark/datasets/ directory:
cd benchmark/datasets
git clone https://github.com/mohammadtavakoli78/BEAM.git
After cloning, benchmark/datasets/BEAM/ should contain chats/, src/, topics/ and other subdirectories.