scriptoriumadmin/ReMe
1
0
Fork 0
mirror of https://github.com/agentscope-ai/ReMe.git synced 2026-11-01 07:03:02 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

feat(benchmark): add read tool to agentic answer and update BEAM results

Browse source 
- Add 'read' to job_tools in BaseAgenticAnswerStep for file reading capability
- Document read tool usage in lme/agentic_answer.yaml system prompt
- Update result-beam.md with latest evaluation scores (OVERALL: 0.623/0.580)
This commit is contained in:
sa-buc 2026-07-23 00:23:44 +08:00
parent ce2e8c193e
commit 5815d84be0
3 changed files with 14 additions and 12 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

22
benchmark/results_md/beam.md
View file

@ -23,14 +23,14 @@
| 题型 | Prompted(limit=15) | Prompted Binary | Agentic | Agentic Binary |
|---|---|---|---|---|
| abstention | 0.464 | 0.464 | 0.514 | 0.514 |
| contradiction_resolution | 0.079 | 0.068 | 0.373 | 0.339 |
| event_ordering | 0.455 | 0.334 | 0.547 | 0.450 |
| information_extraction | 0.653 | 0.589 | 0.818 | 0.764 |
| instruction_following | 0.541 | 0.524 | 0.765 | 0.745 |
| knowledge_update | 0.571 | 0.507 | 0.636 | 0.629 |
| multi_session_reasoning | 0.426 | 0.324 | 0.593 | 0.540 |
| preference_following | 0.718 | 0.676 | 0.838 | 0.824 |
| summarization | 0.516 | 0.303 | 0.661 | 0.478 |
| temporal_reasoning | 0.198 | 0.169 | 0.394 | 0.383 |
| **OVERALL** | **0.462** | **0.396** | **0.614** | **0.567** |
| abstention | 0.464 | 0.464 | 0.464 | 0.464 |
| contradiction_resolution | 0.079 | 0.068 | 0.379 | 0.346 |
| event_ordering | 0.455 | 0.334 | 0.535 | 0.442 |
| information_extraction | 0.653 | 0.589 | 0.795 | 0.758 |
| instruction_following | 0.541 | 0.524 | 0.774 | 0.758 |
| knowledge_update | 0.571 | 0.507 | 0.693 | 0.679 |
| multi_session_reasoning | 0.426 | 0.324 | 0.656 | 0.602 |
| preference_following | 0.718 | 0.676 | 0.805 | 0.796 |
| summarization | 0.516 | 0.303 | 0.672 | 0.504 |
| temporal_reasoning | 0.198 | 0.169 | 0.462 | 0.448 |
| **OVERALL** | **0.462** | **0.396** | **0.623** | **0.580** |

2
reme/steps/benchmark/base/agentic_answer.py
View file

@ -54,7 +54,7 @@ class BaseAgenticAnswerStep(BaseStep):
tool_context_id = f"{self.TOOL_CONTEXT_PREFIX}_{os.getpid()}_local"
wrapper_kwargs = {
"system_prompt": sys_prompt,
"job_tools": ["search", "add_draft", "read_all_draft"],
"job_tools": ["search", "add_draft", "read_all_draft","read"],
"react_config": {"max_iters": self.MAX_ITERATION},
"tool_context_id": tool_context_id,
}

2
reme/steps/benchmark/lme/agentic_answer.yaml
View file

@ -5,6 +5,8 @@ system_prompt: |
## Search Strategy
- You can call 'search' tool to search multiple times (at least once) with different queries to gather comprehensive information.
- Do not stop after the first relevant one unless you are confident that the information is sufficient.
- You can use 'read' tool to read specific parts of a specific file.
## Draft Tool
- Use 'add_draft' to save key findings during search, and 'read_all_draft' to review all saved notes before answering.
## Answer Rules