scriptoriumadmin/ReMe
1
0
Fork 0
mirror of https://github.com/agentscope-ai/ReMe.git synced 2026-11-01 07:03:02 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

docs(benchmark): rebase LME LongMemEval docs onto current main (supersedes #332)

Browse source 
The old PR #332 has been CONFLICTING for ~23 days because its doc layout
changes (docs/en/_toc.yml, docs/en/index.md, docs/zh/_toc.yml,
docs/zh/index.md) referred to files that no longer exist in current main
(the docs layout was refactored to a flat Markdown tree between the time
#332 was authored and today's HEAD). Rather than try to rebase those
obsolete file edits onto a layout that doesn't have them, this PR ports
the genuinely useful content from #332 (the two language-specific
benchmark pages) onto today's main and drops the 4 non-existent file
edits entirely.

Carried over, unchanged in wording:
- docs/en/lme_benchmark.md  (new) — 143 lines covering context_answer_step
  + answer_judge_step inputs/outputs, judge-prompt selection table, job
  wiring example with lme_one_question, and the 4 "notes for benchmark
  authors" footguns
- docs/zh/lme_benchmark.md  (new) — same content in Chinese

Adapted to current layout:
- docs/zh/quick_start.md — still adds the 2-line pointer to the benchmark
  page in the same logical location (right after the auto_* / proactive
  paragraph), but now the link works because the target file is actually
  in docs/zh/
- docs/en/quick_start.md — added the English mirror of the zh line so
  the bilingual quick starts stay symmetric

Explicitly NOT carried over from #332 (since those files don't exist):
- docs/en/_toc.yml, docs/en/index.md Benchmark section + LME card
- docs/zh/_toc.yml, docs/zh/index.md 评测 section + LME card
  (Jupyter-Book toc/index layout is gone; flat docs don't need them.)

If maintainers later want the Benchmark nav group back, that is best
introduced alongside whatever flat-docs navigation scheme replaces the
old Jupyter-Book toc, not in this PR which is scoped to conflict
resolution of the original benchmark prose.

Supersedes #332. Relates to #354 (complete LongMemEval evaluation).
This commit is contained in:
RerankerGuo 2026-07-31 10:09:09 +08:00
parent f3d32e203d
commit c8f2e46464
4 changed files with 286 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

143
docs/en/lme_benchmark.md Normal file
View file

@ -0,0 +1,143 @@
# LongMemEval (LME) Benchmark
ReMe ships two ReMe-native steps that compose into a [LongMemEval](https://github.com/xiaowu0162/longmemeval)-style
benchmark loop: a direct **answer from session context** step, and an **LLM-judge** step that scores that answer
against the dataset's golden answer. Together with the rest of ReMe's auto-memory / search primitives, they let you
plug ReMe into the official LME harness the same way you would any other tool-equipped service.
```{note}
The example below assumes a service started with `reme start config=reme/config/jinli_lme.yaml`. The benchmark loop
itself is not part of `jinli_lme.yaml`; it lives in the LME harness repo. This page only documents the two ReMe-side
steps and their input/output contract.
```
## Steps at a glance
| Step name | What it reads | What it writes |
|---|---|---|
| `context_answer_step` | `query`, `session_context`, `current_date` | `context_answer` |
| `answer_judge_step` | `query`, `agent_answer`, `golden_answer`, `question_type` | `answer_judgement` (`yes` / `no`) |
Both steps require an `agent_wrapper` (the LLM that performs the answering / judging). They are stateless wrappers
over the wrapper's `reply` call — ReMe does not perform any extra retrieval, only formatting.
## `context_answer_step`
Answers a question directly from the supplied session context (i.e. the conversation history for one dialogue
session in the LME dataset). This mirrors the "direct" baseline in the LongMemEval paper: the model sees only the
history, not the agent's memory store.
Required inputs (from `RuntimeContext`):
```yaml
query: string # the question being answered
session_context: string # concatenated history for one session
current_date: string # `YYYY-MM-DD` of "today" so relative phrases resolve
```
Produces:
```yaml
context_answer: string # model's free-form answer
metadata:
query: ...
session_context: ...
current_date: ...
context_answer: ...
```
The default user message template (`user_message`) asks the model to extract relevant facts first and then reason
over them, which matches the public LME prompt.
## `answer_judge_step`
Scores an `agent_answer` against a `golden_answer` with one of four LLM judges. The judge prompt is selected
automatically from `question_type`:
| `question_type` (after normalization) | Judge prompt key |
|---|---|
| `temporal_reasoning` | `temporal_reasoning_system_prompt` |
| `knowledge_update` | `knowledge_update_system_prompt` |
| `single_session_preference` | `single_session_preference_system_prompt` (uses the *preference* rubric user template) |
| anything else | `other_question_types_system_prompt` |
Normalization lowercases the string and replaces `-` and spaces with `_`, so `Temporal-Reasoning`,
`temporal reasoning`, and `TEMPORAL_REASONING` all hit the same judge. Unknown values fall through to the
generic `other_question_types_system_prompt`, which is the right behavior for LME categories other than the
three handled above.
The raw judge output is normalized with a case-insensitive `^\s*(yes|no)\b` match. If that match succeeds, the
final `answer_judgement` is `yes` or `no`; otherwise the lowercased first token of the raw output is kept as-is
(this lets the benchmark harness treat unexpected output as a miss rather than over-reporting).
Required inputs:
```yaml
query: string
agent_answer: string # typically `context_answer` from the previous step
golden_answer: string # from the LME dataset
question_type: string # one of the LME categories above
```
Produces:
```yaml
answer_judgement: string # `yes` / `no` (or raw lowercased token if non-conforming)
metadata:
query: ...
agent_answer: ...
golden_answer: ...
question_type: ...
answer_judgement: ...
raw_answer_judgement: ... # original model output, useful for debugging mismatched judgements
```
## Wiring the steps into a job
The shipped `jinli_lme.yaml` defines only the standard jobs (`update_index`, `auto_memory`, `version`, `search`)
plus a Tokenizer / Embedding / LLM / Agent stack. To make the two benchmark steps callable from the LME harness,
add a job that bundles them. This stanza can sit in a separate YAML or be appended to `jinli_lme.yaml`:
```yaml
jobs:
lme_one_question:
backend: base
description: "Run context_answer_step + answer_judge_step for a single LME question."
parameters:
type: object
properties:
query: {type: string}
session_context: {type: string}
current_date: {type: string}
agent_answer: {type: string}
golden_answer: {type: string}
question_type: {type: string}
required: [query, session_context, agent_answer, golden_answer, question_type]
steps:
- backend: context_answer_step
- backend: answer_judge_step
```
Invoking it from the LME harness is then just another POST request:
```bash
reme app config=jinli_lme.yaml lme_one_question \
query="When did Jon lose his job?" \
session_context="2023-01-19: ..." \
current_date="2024-03-01" \
agent_answer="..." \
golden_answer="..." \
question_type="temporal_reasoning"
```
## Notes for benchmark authors
- Both steps raise `ValueError` for missing required context fields — the harness should treat that as a
per-question failure with `question_id` attached, not as a service-level error.
- `question_type` *must* be normalized on the harness side if the LME dataset emits titles like
`temporal-reasoning`; the step itself only does the safe fallback normalization.
- `answer_judgement` is the field to score against the LME reference scoring script. `raw_answer_judgement` is
kept for diagnostics when the judge model hallucinates a non-yes/no token.
- The benchmark loop should also keep the per-question metadata captured by both steps (especially
`session_context` length and `current_date`) so future time-handling refactors can be re-evaluated against
the same inputs.

3
docs/en/quick_start.md
View file

@ -181,6 +181,9 @@ These flows require a working LLM. Without an LLM configuration, start with basi
For more detail, see [Auto Memory](./auto_memory.md), [Auto Resource](./auto_resource.md),
[Auto Dream](./auto_dream.md), and [Proactive](./proactive.md).
To plug ReMe into benchmarks such as [LongMemEval](https://github.com/xiaowu0162/longmemeval), see
[LME Benchmark](./lme_benchmark.md).
---
## HTTP and Configuration

136
docs/zh/lme_benchmark.md Normal file
View file

@ -0,0 +1,136 @@
# LongMemEval (LME) Benchmark
ReMe 内置两个可以直接拼成 [LongMemEval](https://github.com/xiaowu0162/longmemeval) 评测流程的
Step：一个**从会话上下文直接作答**的 Step，和一个用 LLM 给该答案**打 yes/no 评分**的 Step。配合 ReMe
已有的 `auto_memory`、`search` 等能力，它们可以像其他支持工具的服务一样接入官方 LME harness。
```{note}
下面的示例假定服务以 `reme start config=reme/config/jinli_lme.yaml` 启动。benchmark 循环本身不在
`jinli_lme.yaml` 中，它由 LME harness 仓库负责；本页面只讲清两个 Step 自身的输入输出约定。
```
## 两个 Step 一览
| Step 名称 | 读入 | 写出 |
|---|---|---|
| `context_answer_step` | `query`、`session_context`、`current_date` | `context_answer` |
| `answer_judge_step` | `query`、`agent_answer`、`golden_answer`、`question_type` | `answer_judgement`（`yes` / `no`） |
两个 Step 都依赖 `agent_wrapper`（用于实际回答 / 判分的 LLM）。它们是对 `reply` 的薄封装，不做任何额外
的检索，只负责 prompt 拼装。
## `context_answer_step`
直接根据传入的会话上下文（即一次对话的全部历史）回答问题，对应 LongMemEval 论文中的 "direct" baseline：
模型只看到历史，不接触 agent 的记忆库。
必填字段（来自 `RuntimeContext`）：
```yaml
query: string # 当前问题
session_context: string # 一次会话拼接好的历史
current_date: string # `YYYY-MM-DD` 形式的"今天"，用于解析相对时间
```
产出：
```yaml
context_answer: string # 模型自由形式的回答
metadata:
query: ...
session_context: ...
current_date: ...
context_answer: ...
```
默认的 user 消息模板（`user_message`）会要求模型先抽取相关事实再推理，这与公开的 LME prompt 一致。
## `answer_judge_step`
用四套 LLM 判分 prompt 中的一套，给 `agent_answer` 与 `golden_answer` 打分。判分 prompt 由 `question_type`
自动选择：
| `question_type`（归一化后） | 使用的判分 prompt |
|---|---|
| `temporal_reasoning` | `temporal_reasoning_system_prompt` |
| `knowledge_update` | `knowledge_update_system_prompt` |
| `single_session_preference` | `single_session_preference_system_prompt`（使用 *preference* rubric user 模板） |
| 其它 | `other_question_types_system_prompt` |
归一化规则是 lowercase + 把 `-` 和空格替换成 `_`，因此 `Temporal-Reasoning`、`temporal reasoning`、
`TEMPORAL_REASONING` 都会命中同一个判分 prompt。未知 category 会落到通用
`other_question_types_system_prompt`，这正好对应 LME 中上述三类之外的题目。
判分模型的原始输出会被用大小写不敏感的 `^\s*(yes|no)\b` 正则归一化：命中则 `answer_judgement` 设为
`yes` 或 `no`；否则保留原始输出的首个 token 并 lowercase（这样 harness 可以把意外输出当作失误处理，
而不是被误算成正确）。
必填字段：
```yaml
query: string
agent_answer: string # 一般是上一步的 `context_answer`
golden_answer: string # 来自 LME 数据集
question_type: string # 上述 LME category 之一
```
产出：
```yaml
answer_judgement: string # `yes` / `no`（或未规整时的原 token）
metadata:
query: ...
agent_answer: ...
golden_answer: ...
question_type: ...
answer_judgement: ...
raw_answer_judgement: ... # 模型的原始回答，便于调试判分异常
```
## 接入 Job 的样例
随包发布的 `jinli_lme.yaml` 只配置了常规的 Job（`update_index`、`auto_memory`、`version`、`search`）以及
Tokenizer / Embedding / LLM / Agent 组件栈。要让两个 benchmark Step 能被 LME harness 调用，需要再加一个
Job 把它们串起来。下面这段可以单独放在一个 YAML 里，也可以追加到 `jinli_lme.yaml`：
```yaml
jobs:
lme_one_question:
backend: base
description: "对单条 LME 题目执行 context_answer_step + answer_judge_step。"
parameters:
type: object
properties:
query: {type: string}
session_context: {type: string}
current_date: {type: string}
agent_answer: {type: string}
golden_answer: {type: string}
question_type: {type: string}
required: [query, session_context, agent_answer, golden_answer, question_type]
steps:
- backend: context_answer_step
- backend: answer_judge_step
```
之后从 LME harness 触发就只是一次 POST：
```bash
reme app config=jinli_lme.yaml lme_one_question \
query="When did Jon lose his job?" \
session_context="2023-01-19: ..." \
current_date="2024-03-01" \
agent_answer="..." \
golden_answer="..." \
question_type="temporal_reasoning"
```
## 写 benchmark 时的注意点
- 缺少必填字段时两个 Step 都会抛 `ValueError`，harness 应该把它当作单题失败（带上 `question_id`）处理，
而不是当作服务级错误。
- LME 数据集如果给出的是 `temporal-reasoning` 这种带连字符的 category，harness 侧应自行规一化；Step 内部
只做安全回退处理。
- 评分时应使用 `answer_judgement`，`raw_answer_judgement` 仅用于诊断判分模型输出非 yes/no token 的情形。
- benchmark 循环建议同时保留两个 Step 写出的题目级 metadata（尤其是 `session_context` 长度与
`current_date`），方便后续时间处理逻辑重构后用同一批输入重跑对比。

5
docs/zh/quick_start.md
View file

@ -173,7 +173,10 @@ reme proactive date=2026-06-20
这些流程需要可用 LLM；未配置 LLM 时请先使用 `write/read/search` 这类基础能力。
更多细节见 [Auto Memory](./auto_memory.md)、[Auto Resource](./auto_resource.md)、[Auto Dream](./auto_dream.md) 和
[Proactive](./proactive.md)。
[Proactive](./proactive.md)。
如果是要把 ReMe 接入 [LongMemEval](https://github.com/xiaowu0162/longmemeval) 之类的评测，参考
[LME Benchmark](./lme_benchmark.md)。
---