mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-09-05 08:06:15 +00:00
* chore(benchmark): isolate dataset/workspaces/results per benchmark
- Move shared benchmark/{datasets,memory_workspaces,results} into per-benchmark subdirs benchmark/<name>/{dataset,workspaces,results}
- Update beam/longmemeval config.yaml and run.py path defaults
- Relocate longmemeval download.py to benchmark/longmemeval/ (downloads into dataset/ subdir); inline dataset download docs into README
- Update .gitignore: benchmark/*/{dataset,workspaces,results}/
- Move result-{beam,longmemeval}.md to benchmark/results_md/ and drop result- prefix; update README links
- Fix stale path refs in llm_judge.py and logs/demo_search_format.py
* feat(benchmark): add read tool to agentic answer and update BEAM results
- Add 'read' to job_tools in BaseAgenticAnswerStep for file reading capability
- Document read tool usage in lme/agentic_answer.yaml system prompt
- Update result-beam.md with latest evaluation scores (OVERALL: 0.623/0.580)
* feat(auto_memory): add source line-number markers for note traceability
- Add _format_history hook in AutoMemoryStep with line-number annotation
- Override in BeamAutoMemoryStep to prefix each turn with [Ln] for citation
- Add session_file variable to prompt templates for source marker paths
- Simplify repeated extraction rules by referencing system prompt
- Enhance agentic_answer search strategy (multi-search, read tool hint)
- Add warning log on ReadStep failure
* feat(beam): enhance auto_memory with source markers and pilot ingest tooling
* refactor(beam): rename max_chunk_words to max_segment_words, drop one-off pilot scripts
* feat: add CompressorStep and search_v2 dual-mode session compression
- Add CompressorStep (reme/steps/evolve/compressor.py) for direct LLM
text compression with optional query-guided relevance filtering
- Extend search_v2_step to support query-aware and query-independent
session transcript compression via _compress injected kwargs
- Refactor _source_format.py: split into render_chunk_entries +
join_chunk_entries; session chunks now render line-aligned with
L<n>: prefixes for verbatim/compressed parity
- Add JOB_TOOLS and INJECTED_JOB_KWARGS to BaseAgenticAnswerStep for
per-subclass tool and parameter injection
- LmeAgenticAnswerStep injects _search._compress payload to enable
query-aware compression during benchmark evaluation
- Record compression ablation results in result-longmemeval.md
- Add unit tests for CompressorStep and search compression paths
* refactor(compress): relax session compression to lenient format-preserving strategy and update LME results
* refactor(benchmark): make session compression config-driven via compress_session flag
Move session-transcript compression from LME hard-coded injection to a
runtime context flag set by evaluation.compress_session in each
benchmark config. Compression is off by default for both BEAM and LME,
and BaseAgenticAnswerStep now conditionally injects the _search compress
payload only when the flag is truthy.
* feat(lme/auto_memory): add source attribution markers with line numbers
Add _format_history to annotate each turn with [Ln] line numbers and
expose {session_file} in prompts so the agent can emit bare wikilink-style
source markers like [[session/dialog/s1.jsonl#L1-L2,L5-L6]] at the end
of factual entries. Consolidate the per-prompt body/format rules into
references to the system prompt to avoid drift, and add frontmatter-
protection guidance for the edit tool.
* feat: improve agentic answer prompt and update beam 100K results
- Strengthen abstention rule: prohibit extrapolation from related but
non-direct evidence
- Add multi-angle search after preliminary answer to check for
conflicting/supplementary/updated information
- Add max-iteration fallback to 'Information not found'
- Update beam.md with 100K results (agentscope 2.0.4.post1, from scratch)
including per-type token consumption and memory construction stats
- config.yaml: 100K dataset, 20 workers for BEAM evaluation
- run.py: add memory construction token usage tracking (default agent)
- Overall: 0.635 → 0.654 (+0.019), contradiction_resolution: 0.338 → 0.478
(+0.140), abstention: 0.500 → 0.525 (+0.025)
* feat(read): add session-aware formatting for read tool and update BEAM eval
- Add truncate_session_output in _file_io.py to render jsonl session
lines as [speaker @ time] content before byte-budget truncation
- Add read_step_format_session flag to ReadStep, honoring injected
job kwargs (precedence) and YAML fallback
- Inject read_step_format_session=True into BaseAgenticAnswerStep
so agentic answer reads render session transcripts human-readably
- Refine BEAM agentic_answer prompt: continue multi-angle search
after preliminary answer, forbid fabrication/extrapolation
- Update BEAM config to 1M variant and add sequential 100K-eval /
1M-build shell script
- Refresh benchmark/results_md/beam.md with latest results
* chore(config): disable expand_links in beam and lme search_v2 configs
* refactor(beam): drop one-off sequential 100K-eval-then-1M-build script
* fix(benchmark): add compressor job to beam config and fix BEAM clone instructions
- Add compressor job and compressor as_llm component to reme/config/beam.yaml
(aligned with lme.yaml) so that compress_session: true works for BEAM
- Add graceful degradation guard in search_v2._compress_session_entries:
when the compressor job is missing from the active config, log a warning
and skip compression instead of raising 'Job compressor not found'.
Skipped when there is no app_context so unit tests mocking run_job still
drive compression behavior.
- Fix BEAM download instructions in README.md/README_ZH.md: add mkdir -p
before cd benchmark/beam/dataset (the directory is gitignored and absent
in a fresh clone)
* fix(steps): guard compressor exceptions and fix ReadStep boolean override
1. search_v2: catch per-entry exceptions from run_job('compressor') inside
compress() so asyncio.gather never propagates a compressor failure (e.g.
temporary LLM outage). The failing entry keeps its original body while
remaining entries are still compressed, preserving already-retrieved
search results.
2. read: replace 'context_value or yaml_value' with an existence check so
that a runtime-injected False can explicitly disable a YAML-true
read_step_format_session flag.
Add focused unit tests for both paths.
* fix(search_v2): use existence check for strict_date_filter boolean override
Replace 'context_value or yaml_value' with an existence-based check so
that a runtime-injected False can explicitly disable a YAML-true
strict_date_filter flag, consistent with the read_step_format_session fix.
* refactor(search): simplify strict_date_filter fallback to truthiness-or
* style(test): rename unused param to satisfy pylint W0613
---------
Co-authored-by: sa-buc <jiangniurou.xyf@dail-algo011164204033.ET135>
91 lines
6.8 KiB
YAML
91 lines
6.8 KiB
YAML
compress_prompt: |
|
|
You are a text compression engine. The text below is a conversation transcript. Compress it so it becomes shorter while losing zero information.
|
|
|
|
## Compression Rules
|
|
|
|
1. Delete sentences that carry no real information: greetings, pleasantries, small talk, thanks, apologies, courtesy formulas, filler words, and empty transitions (e.g. "Hi there!", "Hope you're doing well", "Thanks so much!", "As we all know").
|
|
2. You may rewrite, merge, or restructure sentences to make them shorter, but you must NOT lose any information. Every fact, entity, number, date, condition, causal relation, opinion, preference, and nuance present in the original must remain fully recoverable from your output.
|
|
3. Never delete informative content, and never weaken it into a vaguer statement. Compression must come only from removing zero-information phrasing and from tighter wording.
|
|
4. Never add new information, comments, guesses, or interpretations of your own.
|
|
5. Copy critical details verbatim: numbers, dates, times, amounts, identifiers, proper names, quotes, file paths, and code must appear exactly as in the original. Vague or relative time expressions (e.g. "last Wednesday", "two weeks ago") must also be preserved as-is.
|
|
6. Keep the original language of the text, and try to preserve the original format (such as the `[role @ timestamp]` message lines) and the original order of information.
|
|
7. If the entire text carries no real information (pure greetings/small talk), output nothing.
|
|
|
|
## Output Format
|
|
|
|
Output ONLY the compressed text. No explanations, no preamble, no quotes, no markdown fences.
|
|
|
|
# Text
|
|
|
|
{text}
|
|
compress_prompt_zh: |
|
|
你是一个文本压缩引擎。下面的文本是一段对话记录。请压缩它,使其变短,但信息量零损失。
|
|
|
|
## 压缩规则
|
|
|
|
1. 删除没有实际信息量的语句:问候、寒暄、客套、感谢、道歉、口头禅、空洞的过渡语(例如"你好呀!""希望你一切都好""非常感谢!""众所周知")。
|
|
2. 可以改写、合并、调整语句以减小长度,但不得损失任何信息。原文中的每一个事实、实体、数字、日期、条件、因果关系、观点、偏好和细微含义,都必须能从你的输出中完整还原。
|
|
3. 不得删减信息内容,也不得把信息弱化成更模糊的表述。压缩只能来自删除零信息量的表达和更紧凑的措辞。
|
|
4. 不得添加任何新信息、评论、猜测或你自己的解读。
|
|
5. 关键细节必须逐字保留:数字、日期、时间、金额、标识符、专有名词、引语、文件路径和代码必须与原文完全一致。模糊或相对的时间描述(如"上周三""两周前")也必须原样保留。
|
|
6. 保持文本的原始语言,尽量保持原有的格式(如 `[role @ 时间戳]` 的消息行)和信息的原有顺序。
|
|
7. 如果整段文本没有任何实际信息(纯问候/寒暄),则输出空。
|
|
|
|
## 输出格式
|
|
|
|
只输出压缩后的文本。不要解释、不要开场白、不要引号、不要 markdown 代码块。
|
|
|
|
# 文本
|
|
|
|
{text}
|
|
|
|
compress_query_prompt: |
|
|
You are a text compression engine. The text below is a conversation transcript. Compress it so it becomes shorter, guided by the queries as a relevance filter, without losing any potentially useful information.
|
|
|
|
## Compression Rules
|
|
|
|
1. Delete sentences that carry no real information: greetings, pleasantries, small talk, thanks, apologies, courtesy formulas, filler words, and empty transitions (e.g. "Hi there!", "Hope you're doing well", "Thanks so much!", "As we all know").
|
|
2. You may rewrite, merge, or restructure sentences to make them shorter, but the kept content must NOT lose information. Every fact, entity, number, date, condition, causal relation, opinion, preference, and nuance you keep must remain fully recoverable from your output.
|
|
3. Keep ALL content that is possibly or potentially related to ANY of the queries. Even weak, indirect, background, or uncertain relevance to any single query means it MUST be kept.
|
|
4. You may delete a piece of information ONLY when both conditions hold for EVERY query: it is certainly unrelated to that query, AND it is certainly of no help whatsoever for answering that query.
|
|
5. If you are unsure whether something can be deleted, KEEP it. When in doubt, always err on the side of keeping.
|
|
6. Do NOT answer the queries. Do NOT add new information, comments, guesses, or interpretations of your own.
|
|
7. Copy critical details verbatim: numbers, dates, times, amounts, identifiers, proper names, quotes, file paths, and code must appear exactly as in the original. Vague or relative time expressions (e.g. "last Wednesday", "two weeks ago") must also be preserved as-is.
|
|
8. Keep the original language of the text, and try to preserve the original format (such as the `[role @ timestamp]` message lines) and the original order of information.
|
|
|
|
## Output Format
|
|
|
|
Output ONLY the compressed text. No explanations, no preamble, no quotes, no markdown fences.
|
|
|
|
# Queries
|
|
|
|
{queries}
|
|
|
|
# Text
|
|
|
|
{text}
|
|
compress_query_prompt_zh: |
|
|
你是一个文本压缩引擎。下面的文本是一段对话记录。请以下列 query 列表作为相关性过滤条件,压缩文本使其变短,同时不丢失任何可能有用的信息。
|
|
|
|
## 压缩规则
|
|
|
|
1. 删除没有实际信息量的语句:问候、寒暄、客套、感谢、道歉、口头禅、空洞的过渡语(例如"你好呀!""希望你一切都好""非常感谢!""众所周知")。
|
|
2. 可以改写、合并、调整语句以减小长度,但保留下来的内容不得损失信息。你保留的每一个事实、实体、数字、日期、条件、因果关系、观点、偏好和细微含义,都必须能从你的输出中完整还原。
|
|
3. 保留任何可能、潜在与任意一个 query 相关的内容。哪怕只与其中某一个 query 相关性微弱、间接、只是背景信息或不确定,也必须保留。
|
|
4. 只有对每一个 query 都同时满足以下两个条件时,才可以删除一条信息:它一定与该 query 无关,并且对回答该 query 一定没有任何帮助。
|
|
5. 对于不确定是否可以删除的信息,一律保留。拿不准时,永远倾向于保留。
|
|
6. 不要回答这些 query 本身。不得添加任何新信息、评论、猜测或你自己的解读。
|
|
7. 关键细节必须逐字保留:数字、日期、时间、金额、标识符、专有名词、引语、文件路径和代码必须与原文完全一致。模糊或相对的时间描述(如"上周三""两周前")也必须原样保留。
|
|
8. 保持文本的原始语言,尽量保持原有的格式(如 `[role @ 时间戳]` 的消息行)和信息的原有顺序。
|
|
|
|
## 输出格式
|
|
|
|
只输出压缩后的文本。不要解释、不要开场白、不要引号、不要 markdown 代码块。
|
|
|
|
# Query 列表
|
|
|
|
{queries}
|
|
|
|
# 文本
|
|
|
|
{text}
|