mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-11-19 09:51:51 +00:00
feat(benchmark): add MobileMem reproduction
This commit is contained in:
parent
c1de31ab2c
commit
bdbf299406
22 changed files with 1725 additions and 0 deletions
79
benchmark/mobilemem/README.md
Normal file
79
benchmark/mobilemem/README.md
Normal file
|
|
@ -0,0 +1,79 @@
|
|||
# MobileMem-Omni Visual Reasoning
|
||||
|
||||
Evaluate ReMe Image on the published `visual_reasoning` questions using each user's complete chronological
|
||||
dialogue history. Original dialogue text, including any descriptions already in it, is retained alongside images.
|
||||
Separate summaries/captions, reference answers, evidence annotations and question types are not added to candidate Jobs.
|
||||
Native `multi_visual_reasoning` questions are not relabelled or included.
|
||||
|
||||
## Run
|
||||
|
||||
Install both plugins and extract the official archive without changing its images:
|
||||
|
||||
```bash
|
||||
pip install -e ".[as]" -e plugins/mobilemem -e plugins/mobilemem-judge
|
||||
mkdir -p /path/to/omni/image
|
||||
python -m zipfile -e /path/to/omni/image.zip /path/to/omni/image
|
||||
export LLM_API_KEY="..."
|
||||
export LLM_BASE_URL="..."
|
||||
python -m benchmark.mobilemem.run --dataset /path/to/omni --output /path/to/exp-xxxx
|
||||
```
|
||||
|
||||
[config.yaml](config.yaml) supplies defaults: `qwen3.8-max`, four concurrent users and 16 concurrent questions.
|
||||
Override them with `--model`, `--user-concurrency` and `--query-concurrency`.
|
||||
Every run creates `<output>/mobilemem/<model>` and rejects an existing directory.
|
||||
|
||||
## Evaluation
|
||||
|
||||
Each user has a fresh workspace. All source sessions are ingested chronologically through multimodal
|
||||
`auto_memory`, then indexed once. Every question starts a fresh 30-iteration `agentic_answer` Agent with
|
||||
BM25 `search`, `read` and `read_image`. Search covers daily notes, digest notes and original session text.
|
||||
Original Image IDs accompany source images; saved native attachments can be reopened by the answer Agent.
|
||||
|
||||
The [candidate preset](../../plugins/mobilemem/src/reme_mobilemem/benchmark.yaml) uses the same model for memory,
|
||||
answering and compression: temperature 0, thinking enabled, 65,536 output tokens, 1,000,000-token context and
|
||||
at most 4,096 images. Native context ratios remain 0.8/0.1. Input images are not resized; MPO inputs retain
|
||||
the existing primary-picture PNG conversion. Published path aliases are resolved by unique image basename.
|
||||
|
||||
The official QA template wraps the question with evidence-tool instructions. Only the isolated
|
||||
[MobileMem Judge](../../plugins/mobilemem-judge/README.md) receives references and evidence. Its default is the
|
||||
MemBase executable-default `mobilemem-omni-judge` rubric, `qwen3.8-max`, temperature 0, thinking disabled and
|
||||
4,096 output tokens. Scores report binary LLM-Judge accuracy and the official token-overlap F1 and BLEU-1.
|
||||
|
||||
## Results
|
||||
|
||||
Each user's `workspace/` retains notes, source JSONL, images and native AgentScope trajectories.
|
||||
`sessions/*.json` and `questions/*/` retain responses, judgments, metrics and technical errors.
|
||||
`summary.json` reports completed/expected questions, scores, memory/answer token totals and averages.
|
||||
Provider usage includes compression; absent cache fields stay unknown. Judge usage is excluded from method cost.
|
||||
Native interrupted replies and failed Jobs remain unscored technical gaps. The runner adds no fallback or retry queue.
|
||||
|
||||
Sources: [MobileMem evaluation code](https://github.com/zjunlp/MobileMem/tree/fa5cd13c9c659699df9caf650987ba2bea8ca085/omni/eval/eval)
|
||||
and [MemBase judge](https://github.com/zjunlp/MemBase/blob/a9ba9495d48541d3e68588ff097294cc7a1201bc/membase/inference_utils/prompts.py).
|
||||
Dataset revision: `989ba00900bddf4e0d4b0c7dd52f0840a7ee3a54`.
|
||||
|
||||
## Upstream code license
|
||||
|
||||
The QA template and overlap metrics in `run.py` are adapted from ZJUNLP MobileMem's evaluation code at
|
||||
`fa5cd13c9c659699df9caf650987ba2bea8ca085`, under the following license.
|
||||
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 ZJUNLP
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
48
benchmark/mobilemem/README_ZH.md
Normal file
48
benchmark/mobilemem/README_ZH.md
Normal file
|
|
@ -0,0 +1,48 @@
|
|||
# MobileMem-Omni 视觉推理评测
|
||||
|
||||
使用每个用户完整、按时间排序的原始对话,评测 ReMe Image 在官方 `visual_reasoning` 题目上的表现。
|
||||
记忆与回答 Job 仅接收原始文本和图像,不接收数据集摘要、caption、标准答案、证据标注或题型。
|
||||
官方 `multi_visual_reasoning` 不被重命名,也不纳入这个评测范围。
|
||||
|
||||
## 运行
|
||||
|
||||
安装两个插件,并按原目录结构解压官方图片:
|
||||
|
||||
```bash
|
||||
pip install -e ".[as]" -e plugins/mobilemem -e plugins/mobilemem-judge
|
||||
mkdir -p /path/to/omni/image
|
||||
python -m zipfile -e /path/to/omni/image.zip /path/to/omni/image
|
||||
export LLM_API_KEY="..."
|
||||
export LLM_BASE_URL="..."
|
||||
python -m benchmark.mobilemem.run --dataset /path/to/omni --output /path/to/exp-xxxx
|
||||
```
|
||||
|
||||
[config.yaml](config.yaml) 默认使用 `qwen3.8-max`、4 个用户并发、16 个题目并发;
|
||||
可通过 `--model`、`--user-concurrency`、`--query-concurrency` 覆盖。
|
||||
每次运行创建 `<output>/mobilemem/<model>`,已有目录不会被覆盖。
|
||||
|
||||
## 评测设置
|
||||
|
||||
每个用户使用独立的新工作空间,按时间顺序通过多模态 `auto_memory` 摄入全部 session,然后更新一次索引。
|
||||
每题使用新的 `agentic_answer` Agent,最多 30 轮,工具为 BM25 `search`、`read` 和 `read_image`。
|
||||
检索范围包含 daily、digest 笔记及原始 session 正文;保存的原生图像附件与 Image ID 可供回答 Agent 重新读图。
|
||||
|
||||
[候选模型配置](../../plugins/mobilemem/src/reme_mobilemem/benchmark.yaml) 在记忆、回答和压缩阶段使用同一模型:
|
||||
temperature 为 0,开启 thinking,输出预算 65,536,上下文为 1,000,000 token,最多 4,096 张图。
|
||||
原生上下文比例保持 0.8/0.1。输入图片不缩放;MPO 保持既有的主图 PNG 转换。
|
||||
数据集图片路径别名通过唯一 basename 对应原件。
|
||||
|
||||
问题使用官方 QA 模板,并提示通过证据工具作答。标准答案与证据仅交给独立的
|
||||
[MobileMem Judge](../../plugins/mobilemem-judge/README.md)。默认采用 MemBase 可执行默认
|
||||
`mobilemem-omni-judge` 评分模板,模型为 `qwen3.8-max`,temperature 为 0,关闭 thinking,输出预算 4,096。
|
||||
报告包含二元 LLM-Judge 准确率及官方 token-overlap F1、BLEU-1。
|
||||
|
||||
## 结果
|
||||
|
||||
每个用户的 `workspace/` 保存笔记、原始 JSONL、图像及原生 AgentScope 轨迹。
|
||||
`sessions/*.json` 与 `questions/*/` 保存响应、评分、指标和技术错误。
|
||||
`summary.json` 汇总完成/应测题数、指标、记忆与回答 token 总量及平均值。
|
||||
用量覆盖 provider 返回的压缩调用;缺失的 cache 字段保持未知。Judge 用量不计入方法成本。
|
||||
原生中断与失败 Job 作为未评分的技术缺口保存;runner 不额外加入 fallback 或重试队列。
|
||||
|
||||
协议来源、数据版本和改编代码的完整 MIT 许可证见 [英文说明](README.md)。
|
||||
3
benchmark/mobilemem/config.yaml
Normal file
3
benchmark/mobilemem/config.yaml
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
model: qwen3.8-max
|
||||
user_concurrency: 4
|
||||
query_concurrency: 16
|
||||
518
benchmark/mobilemem/run.py
Normal file
518
benchmark/mobilemem/run.py
Normal file
|
|
@ -0,0 +1,518 @@
|
|||
"""Evaluate ReMe Image on MobileMem-Omni Visual Reasoning.
|
||||
|
||||
Official QA and overlap metrics: Copyright (c) 2026 ZJUNLP, MIT License.
|
||||
The complete upstream notice is retained in README.md.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import base64
|
||||
from collections import Counter
|
||||
from contextlib import contextmanager
|
||||
from contextvars import ContextVar
|
||||
from datetime import datetime
|
||||
import io
|
||||
import json
|
||||
import math
|
||||
import mimetypes
|
||||
from pathlib import Path, PurePosixPath
|
||||
import re
|
||||
|
||||
from PIL import Image, ImageOps, JpegImagePlugin
|
||||
import yaml
|
||||
|
||||
from reme import Application
|
||||
from reme.config import resolve_app_config
|
||||
|
||||
|
||||
def timestamp(value: str) -> str:
|
||||
"""Preserve published start-time precision in AgentScope ISO timestamps."""
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00")).isoformat()
|
||||
|
||||
|
||||
def load_mobilemem(root: Path) -> list[dict]:
|
||||
"""Keep complete raw trajectories, selecting only the native visual category."""
|
||||
root = root.expanduser().resolve()
|
||||
records = []
|
||||
for name in ("data.jsonl", "questions.jsonl"):
|
||||
rows = {}
|
||||
with (root / name).open(encoding="utf-8") as stream:
|
||||
for line in stream:
|
||||
if line.strip():
|
||||
row = json.loads(line)
|
||||
identity = str(row["uuid"])
|
||||
if identity in rows:
|
||||
raise ValueError(f"Duplicate MobileMem user: {identity}")
|
||||
rows[identity] = row
|
||||
records.append(rows)
|
||||
histories, questions = records
|
||||
if set(histories) != set(questions):
|
||||
raise ValueError("Every MobileMem user must have a trajectory and question record")
|
||||
return [
|
||||
{
|
||||
"id": f"uuid_{identity}",
|
||||
"image_root": root / "image",
|
||||
"sessions": sorted(
|
||||
row["sessions"],
|
||||
key=lambda session: datetime.fromisoformat(timestamp(session["event_start_time"])),
|
||||
),
|
||||
"questions": [
|
||||
question
|
||||
for question in questions[identity]["questions"]
|
||||
if question["question_type"] == "visual_reasoning"
|
||||
],
|
||||
}
|
||||
for identity, row in histories.items()
|
||||
]
|
||||
|
||||
|
||||
def resolve_image(root: Path, source: str, aliases: dict) -> Path:
|
||||
"""Resolve exact paths or unique published basename aliases inside image/."""
|
||||
root = root.resolve()
|
||||
relative = PurePosixPath(source)
|
||||
if relative.is_absolute() or ".." in relative.parts or "\\" in source or ":" in source:
|
||||
raise ValueError(f"Invalid MobileMem image path: {source}")
|
||||
parts = relative.parts[1:] if relative.parts[0] == "image" else relative.parts
|
||||
target = (root / Path(*parts)).resolve()
|
||||
if not target.is_relative_to(root):
|
||||
raise ValueError(f"MobileMem image escapes image/: {source}")
|
||||
if target.is_file():
|
||||
return target
|
||||
if not aliases:
|
||||
for path in root.rglob("*"):
|
||||
if path.is_file() and path.resolve().is_relative_to(root):
|
||||
if (mimetypes.guess_type(path)[0] or "").startswith("image/"):
|
||||
aliases.setdefault(path.name, []).append(path.resolve())
|
||||
matches = aliases.get(relative.name, [])
|
||||
if len(matches) != 1:
|
||||
raise ValueError(f"Missing or ambiguous MobileMem image: {source}")
|
||||
return matches[0]
|
||||
|
||||
|
||||
def image_block(path: Path) -> dict:
|
||||
"""Use original bytes, or the existing lossless MPO primary-picture conversion."""
|
||||
data = path.read_bytes()
|
||||
if len(data) > 50 * 1024 * 1024:
|
||||
raise ValueError(f"Oversized MobileMem image: {path.name}")
|
||||
opener = JpegImagePlugin.JpegImageFile if data.startswith(b"\xff\xd8\xff") else Image.open
|
||||
with opener(io.BytesIO(data)) as image:
|
||||
mime, mpo = image.get_format_mimetype(), "mp" in image.info
|
||||
if not mime or not mime.startswith("image/") or image.width * image.height > 50_000_000:
|
||||
raise ValueError(f"Invalid or oversized MobileMem image: {path.name}")
|
||||
image.verify()
|
||||
with opener(io.BytesIO(data)) as image:
|
||||
image.load()
|
||||
if mpo:
|
||||
with ImageOps.exif_transpose(image) as oriented, oriented.convert("RGB") as picture:
|
||||
buffer = io.BytesIO()
|
||||
picture.save(buffer, format="PNG")
|
||||
data, mime = buffer.getvalue(), "image/png"
|
||||
return {
|
||||
"type": "data",
|
||||
"source": {
|
||||
"type": "base64",
|
||||
"data": base64.b64encode(data).decode("ascii"),
|
||||
"media_type": mime,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def session_messages(session: dict, image_root: Path, aliases=None) -> list[dict]:
|
||||
"""Convert original dialogue directly; annotations never enter the candidate."""
|
||||
aliases = {} if aliases is None else aliases
|
||||
messages = []
|
||||
for turn in session["dialogue"]:
|
||||
content = [{"type": "text", "text": turn.get("content", "")}]
|
||||
refs = turn.get("image_inline") or []
|
||||
refs = [refs] if isinstance(refs, str) else refs
|
||||
if turn["content_type"] == "image" and not refs:
|
||||
raise ValueError("MobileMem image turn has no image reference")
|
||||
for source in refs:
|
||||
content.extend(
|
||||
[
|
||||
{"type": "text", "text": f"[Image ID: {source}]"},
|
||||
image_block(resolve_image(image_root, source, aliases)),
|
||||
],
|
||||
)
|
||||
messages.append(
|
||||
{
|
||||
"id": f"{session['session_id']}:{turn['turn']}",
|
||||
"name": turn["role"],
|
||||
"role": turn["role"],
|
||||
"content": content,
|
||||
"created_at": timestamp(session["event_start_time"]),
|
||||
},
|
||||
)
|
||||
return messages
|
||||
|
||||
|
||||
def evidence_values(question: dict) -> list[str]:
|
||||
"""Keep the official Raw2Locomo/MemBase evidence-string conversion."""
|
||||
return [
|
||||
str(item["explanation"]) if isinstance(item, dict) else item
|
||||
for item in question.get("evidence", [])
|
||||
if (isinstance(item, dict) and item.get("explanation")) or isinstance(item, str)
|
||||
]
|
||||
|
||||
|
||||
ANSWER_RULES = (
|
||||
"Answer Rules:\n"
|
||||
"1. If options are provided in the question, use the original option text as your answer.\n"
|
||||
"2. For multiple-choice questions (multiple-select), select all correct options.\n"
|
||||
"3. For open-ended questions, respond freely."
|
||||
)
|
||||
|
||||
|
||||
def query_prompt(question: str, context: str) -> str:
|
||||
"""Use the published Text Memory QA template verbatim."""
|
||||
return (
|
||||
f"Question:\n{question}\nPlease answer the question based on the following memories:\n"
|
||||
f"{context}\n{ANSWER_RULES}"
|
||||
)
|
||||
|
||||
|
||||
def tokenize(text: str) -> list[str]:
|
||||
"""Official lowercase ASCII word and individual CJK-character tokens."""
|
||||
text = re.sub(r"[\u0000-\u002F\u003A-\u0040\u005B-\u0060\u007B-\u007E]", " ", str(text).lower())
|
||||
return re.findall(r"[\u4e00-\u9fff]|[a-z0-9]+", text)
|
||||
|
||||
|
||||
def score_answer(reference: str, prediction: str) -> dict[str, float]:
|
||||
"""Official token-overlap F1 and BLEU-1."""
|
||||
pt, rt = tokenize(prediction), tokenize(reference)
|
||||
overlap = sum((Counter(pt) & Counter(rt)).values())
|
||||
f1 = 1.0 if not pt and not rt else 0.0
|
||||
if pt and rt and overlap:
|
||||
precision, recall = overlap / len(pt), overlap / len(rt)
|
||||
f1 = 2 * precision * recall / (precision + recall)
|
||||
bleu = overlap / len(pt) * (math.exp(1 - len(rt) / len(pt)) if len(pt) < len(rt) else 1) if pt else 0.0
|
||||
return {"f1": f1, "bleu_1": bleu}
|
||||
|
||||
|
||||
_CURRENT = ContextVar("mobilemem_usage", default=None)
|
||||
_KEYS = (
|
||||
"input_tokens",
|
||||
"output_tokens",
|
||||
"total_tokens",
|
||||
"cached_input_tokens",
|
||||
"noncached_input_tokens",
|
||||
"reasoning_tokens",
|
||||
)
|
||||
|
||||
|
||||
def normalized_usage(raw: dict) -> dict:
|
||||
"""Keep absent cache fields unknown and avoid double-counting cache subsets."""
|
||||
incoming = raw.get("input_tokens", raw.get("prompt_tokens"))
|
||||
outgoing = raw.get("output_tokens", raw.get("completion_tokens"))
|
||||
total = raw.get("total_tokens")
|
||||
if total is None and incoming is not None and outgoing is not None:
|
||||
total = incoming + outgoing
|
||||
cached = raw.get(
|
||||
"cached_input_tokens",
|
||||
raw.get("cached_tokens", (raw.get("prompt_tokens_details") or {}).get("cached_tokens")),
|
||||
)
|
||||
return {
|
||||
"input_tokens": incoming,
|
||||
"output_tokens": outgoing,
|
||||
"total_tokens": total,
|
||||
"cached_input_tokens": cached,
|
||||
"noncached_input_tokens": incoming - cached if incoming is not None and cached is not None else None,
|
||||
"reasoning_tokens": raw.get(
|
||||
"reasoning_tokens",
|
||||
(raw.get("completion_tokens_details") or {}).get("reasoning_tokens"),
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def aggregate_usage(records: list[dict]) -> dict:
|
||||
"""Return totals and coverage; unavailable counters remain unknown."""
|
||||
return {
|
||||
key: (
|
||||
sum(row[key] for row in records if row.get(key) is not None)
|
||||
if any(row.get(key) is not None for row in records)
|
||||
else None
|
||||
)
|
||||
for key in _KEYS
|
||||
} | {
|
||||
"records": sum(row.get("records", 1) for row in records),
|
||||
"coverage": {
|
||||
key: sum(row.get("coverage", {}).get(key, int(row.get(key) is not None)) for row in records)
|
||||
for key in _KEYS
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@contextmanager
|
||||
def usage_scope():
|
||||
"""Attribute each successful request to its invoking async task."""
|
||||
records = []
|
||||
token = _CURRENT.set(records)
|
||||
try:
|
||||
yield records
|
||||
finally:
|
||||
_CURRENT.reset(token)
|
||||
|
||||
|
||||
def observe_usage(app) -> None:
|
||||
"""Observe lifecycle-owned non-streaming clients, including the compressor."""
|
||||
for component in app.context.components.get("as_llm", {}).values():
|
||||
client = getattr(getattr(component.model, "client", None), "_client", None)
|
||||
if client is None or getattr(client, "_mobilemem_usage_observer", False):
|
||||
continue
|
||||
original = client.send
|
||||
|
||||
async def send(request, *args, _send=original, **kwargs):
|
||||
response = await _send(request, *args, **kwargs)
|
||||
records = _CURRENT.get()
|
||||
if records is not None and response.status_code == 200 and response.is_stream_consumed:
|
||||
try:
|
||||
raw = response.json()
|
||||
except (ValueError, UnicodeError):
|
||||
raw = {}
|
||||
usage = raw.get("usage") if isinstance(raw, dict) else None
|
||||
records.append(normalized_usage(usage if isinstance(usage, dict) else {}))
|
||||
return response
|
||||
|
||||
client.send = send
|
||||
client._mobilemem_usage_observer = True # pylint: disable=protected-access
|
||||
|
||||
|
||||
def observe_completion(app) -> None:
|
||||
"""Keep native interrupted replies unscored; successful replies are unchanged."""
|
||||
for wrapper in app.context.components.get("agent_wrapper", {}).values():
|
||||
original = wrapper.reply
|
||||
|
||||
async def reply(inputs, *args, _reply=original, **kwargs):
|
||||
result = await _reply(inputs, *args, **kwargs)
|
||||
last = result.get("last_message") or {}
|
||||
if str(last.get("finished_reason")) == "interrupted" or last.get("error") is not None:
|
||||
raise RuntimeError("Native Agent was interrupted; its state remains saved.")
|
||||
return result
|
||||
|
||||
wrapper.reply = reply
|
||||
|
||||
|
||||
def save(path: Path, value) -> None:
|
||||
"""Write an evaluation artifact under the newly created output directory."""
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(value, ensure_ascii=False, indent=2, default=str) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def candidate_config(workspace: Path, model: str) -> dict:
|
||||
"""Select one candidate model; behavioral defaults belong to the plugin preset."""
|
||||
config = resolve_app_config(
|
||||
config="mobilemem-benchmark",
|
||||
plugins=["mobilemem"],
|
||||
workspace_dir=str(workspace),
|
||||
log_config=False,
|
||||
enable_logo=False,
|
||||
log_to_console=False,
|
||||
log_to_file=True,
|
||||
components={"as_llm": {name: {"model": model} for name in ("default", "bench", "compressor")}},
|
||||
)
|
||||
config["components"].pop("as_embedding", None)
|
||||
config["components"].pop("embedding_store", None)
|
||||
return config
|
||||
|
||||
|
||||
def judge_config(workspace: Path) -> dict:
|
||||
"""Keep scoring separate from candidate memory, model and evidence tools."""
|
||||
return {
|
||||
"plugins": ["mobilemem-judge"],
|
||||
"service": {"backend": "http"},
|
||||
"workspace_dir": str(workspace),
|
||||
"enable_logo": False,
|
||||
"log_to_console": False,
|
||||
"log_to_file": True,
|
||||
}
|
||||
|
||||
|
||||
def check_response(response):
|
||||
"""Keep failed Jobs as technical errors rather than incorrect predictions."""
|
||||
if not response.success:
|
||||
raise RuntimeError("ReMe Job failed; see the saved response artifact.")
|
||||
return response
|
||||
|
||||
|
||||
async def evaluate_user(scene, directory: Path, model: str, judge, query_slots) -> list[dict]:
|
||||
"""Ingest the complete chronological history into one fresh user workspace."""
|
||||
app = Application(**candidate_config(directory / "workspace", model))
|
||||
results, memory_usage, aliases = [], [], {}
|
||||
try:
|
||||
await app.start()
|
||||
observe_usage(app)
|
||||
observe_completion(app)
|
||||
for ordinal, session in enumerate(scene["sessions"]):
|
||||
with usage_scope() as measured:
|
||||
response = await app.run_job(
|
||||
"auto_memory",
|
||||
messages=await asyncio.to_thread(session_messages, session, scene["image_root"], aliases),
|
||||
session_id=session["session_id"],
|
||||
date=timestamp(session["event_start_time"])[:10],
|
||||
include_images=True,
|
||||
include_source_sessions=True,
|
||||
)
|
||||
save(
|
||||
directory / "sessions" / f"{ordinal:05d}.json",
|
||||
{
|
||||
"session_id": session["session_id"],
|
||||
"response": response.model_dump(),
|
||||
"usage": aggregate_usage(measured),
|
||||
},
|
||||
)
|
||||
check_response(response)
|
||||
memory_usage.extend(measured)
|
||||
print(f"{scene['id']}: session {ordinal + 1}/{len(scene['sessions'])}", flush=True)
|
||||
check_response(await app.run_job("index_update"))
|
||||
save(directory / "memory_usage.json", aggregate_usage(memory_usage))
|
||||
|
||||
async def answer_question(ordinal, question):
|
||||
question_dir = directory / "questions" / f"{ordinal:04d}"
|
||||
try:
|
||||
async with query_slots:
|
||||
with usage_scope() as measured:
|
||||
response = await app.run_job(
|
||||
"agentic_answer",
|
||||
query=query_prompt(question["question"], "Use the available evidence tools."),
|
||||
scene_id=scene["id"],
|
||||
include_images=True,
|
||||
include_source_sessions=True,
|
||||
max_iters=30,
|
||||
)
|
||||
answer_record = {
|
||||
"question_id": question["question_id"],
|
||||
"response": response.model_dump(),
|
||||
"usage": aggregate_usage(measured),
|
||||
}
|
||||
save(question_dir / "answer-response.json", answer_record)
|
||||
check_response(response)
|
||||
save(question_dir / "answer.json", answer_record)
|
||||
prediction = response.answer
|
||||
judgment = await judge.run_job(
|
||||
"answer_judge",
|
||||
question=question["question"],
|
||||
reference=question["answer"],
|
||||
prediction=prediction,
|
||||
evidence=evidence_values(question),
|
||||
)
|
||||
save(question_dir / "judge.json", judgment.model_dump())
|
||||
check_response(judgment)
|
||||
metrics = score_answer(question["answer"], prediction)
|
||||
metrics["llm_judge"] = judgment.metadata["llm_judge_score"]
|
||||
result = {
|
||||
"question_id": question["question_id"],
|
||||
"category": question["question_type"],
|
||||
"question": question["question"],
|
||||
"reference": question["answer"],
|
||||
"prediction": prediction,
|
||||
"metrics": metrics,
|
||||
"usage": aggregate_usage(measured),
|
||||
"judge_parse_default_zero": judgment.metadata["parse_default_zero"],
|
||||
}
|
||||
save(question_dir / "result.json", result)
|
||||
results.append(result)
|
||||
print(f"{scene['id']}: {question['question_id']} completed", flush=True)
|
||||
except Exception as error: # pylint: disable=broad-exception-caught
|
||||
save(question_dir / "error.json", {"error_type": type(error).__name__})
|
||||
print(f"{scene['id']}: {question['question_id']} failed ({type(error).__name__})", flush=True)
|
||||
|
||||
await asyncio.gather(*(answer_question(i, question) for i, question in enumerate(scene["questions"])))
|
||||
except Exception as error: # pylint: disable=broad-exception-caught
|
||||
save(directory / "error.json", {"error_type": type(error).__name__})
|
||||
print(f"{scene['id']}: ingestion failed ({type(error).__name__})", flush=True)
|
||||
finally:
|
||||
await app.close()
|
||||
save(directory / "summary.json", {"completed": len(results), "expected": len(scene["questions"])})
|
||||
return results
|
||||
|
||||
|
||||
async def run(scenes, output: Path, model: str, user_concurrency: int, query_concurrency: int) -> bool:
|
||||
"""Run independent users concurrently and save official metrics plus method cost."""
|
||||
judge = Application(**judge_config(output / "judge-workspace"))
|
||||
rows = []
|
||||
user_slots, query_slots = asyncio.Semaphore(user_concurrency), asyncio.Semaphore(query_concurrency)
|
||||
try:
|
||||
await judge.start()
|
||||
|
||||
async def user_job(scene):
|
||||
async with user_slots:
|
||||
rows.extend(await evaluate_user(scene, output / "users" / scene["id"], model, judge, query_slots))
|
||||
|
||||
await asyncio.gather(*(user_job(scene) for scene in scenes))
|
||||
finally:
|
||||
await judge.close()
|
||||
expected = sum(len(scene["questions"]) for scene in scenes)
|
||||
memory = [json.loads(path.read_text(encoding="utf-8")) for path in output.glob("users/*/memory_usage.json")]
|
||||
answers = [json.loads(path.read_text(encoding="utf-8")) for path in output.glob("users/*/questions/*/answer.json")]
|
||||
answer_usage = [row["usage"] for row in answers]
|
||||
total = aggregate_usage(memory + answer_usage)
|
||||
summary = {
|
||||
"model": model,
|
||||
"completed": len(rows),
|
||||
"expected": expected,
|
||||
"metrics": {
|
||||
key: sum(row["metrics"][key] for row in rows) / len(rows) if rows else None
|
||||
for key in ("f1", "bleu_1", "llm_judge")
|
||||
},
|
||||
"memory_usage": aggregate_usage(memory),
|
||||
"answer_usage": aggregate_usage(answer_usage),
|
||||
"token_usage": total,
|
||||
"mean_tokens_per_user": {
|
||||
key: total[key] / len(memory) if memory and total[key] is not None else None
|
||||
for key in (
|
||||
"input_tokens",
|
||||
"output_tokens",
|
||||
"total_tokens",
|
||||
"cached_input_tokens",
|
||||
"noncached_input_tokens",
|
||||
)
|
||||
},
|
||||
"mean_tokens_per_question": {
|
||||
key: total[key] / len(answers) if answers and total[key] is not None else None
|
||||
for key in (
|
||||
"input_tokens",
|
||||
"output_tokens",
|
||||
"total_tokens",
|
||||
"cached_input_tokens",
|
||||
"noncached_input_tokens",
|
||||
)
|
||||
},
|
||||
"judge_parse_default_zero": sum(row["judge_parse_default_zero"] for row in rows),
|
||||
}
|
||||
save(output / "summary.json", summary)
|
||||
print(json.dumps(summary, ensure_ascii=False, indent=2), flush=True)
|
||||
return len(rows) == expected
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Create a fresh evaluation; the PR reproduces ReMe Image only."""
|
||||
defaults = yaml.safe_load(Path(__file__).with_name("config.yaml").read_text(encoding="utf-8"))
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--dataset", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--model", default=defaults["model"])
|
||||
parser.add_argument("--user-concurrency", type=int, default=defaults["user_concurrency"])
|
||||
parser.add_argument("--query-concurrency", type=int, default=defaults["query_concurrency"])
|
||||
args = parser.parse_args()
|
||||
if min(args.user_concurrency, args.query_concurrency) < 1:
|
||||
parser.error("Concurrency must be positive")
|
||||
scenes = load_mobilemem(args.dataset)
|
||||
output = args.output.expanduser().resolve() / "mobilemem" / args.model
|
||||
output.mkdir(parents=True, exist_ok=False)
|
||||
save(
|
||||
output / "settings.json",
|
||||
{
|
||||
"dataset": str(args.dataset.expanduser().resolve()),
|
||||
"model": args.model,
|
||||
"candidate": "mobilemem-benchmark",
|
||||
"judge": "mobilemem-judge",
|
||||
"question_type": "visual_reasoning",
|
||||
"user_concurrency": args.user_concurrency,
|
||||
"query_concurrency": args.query_concurrency,
|
||||
},
|
||||
)
|
||||
return 0 if asyncio.run(run(scenes, output, args.model, args.user_concurrency, args.query_concurrency)) else 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
21
plugins/mobilemem-judge/LICENSE-CODE
Normal file
21
plugins/mobilemem-judge/LICENSE-CODE
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
MIT License
|
||||
|
||||
Copyright (c) 2026 ZJUNLP
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
25
plugins/mobilemem-judge/README.md
Normal file
25
plugins/mobilemem-judge/README.md
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
# ReMe MobileMem Judge
|
||||
|
||||
Independent `answer_judge` Job for saved MobileMem-Omni predictions. Install with
|
||||
`pip install -e plugins/mobilemem-judge` and enable `mobilemem-judge` in a separate
|
||||
evaluation application/workspace, not the candidate memory application.
|
||||
|
||||
Inputs are `question`, `reference`, `prediction`, and optional `evidence` (a list
|
||||
of original evidence values). `judge_protocol` defaults to `membase_default`, the
|
||||
executable MemBase rubric used by this benchmark. `native_strict` selects the
|
||||
separately published strict rubric. Their protocol identifiers remain distinct.
|
||||
|
||||
Both modes retain the official input interpolation, WRONG-first verdict parsing,
|
||||
and default-zero handling of successful responses without a valid label.
|
||||
`response.metadata` contains `llm_judge_score`, `label`, `parse_default_zero`,
|
||||
`default_judge_protocol`, `prompt`, `raw_response`, and `usage`. Provider errors
|
||||
remain failed Jobs. Judge usage is evaluation cost, not method cost.
|
||||
|
||||
Defaults are `qwen3.8-max`, temperature 0, thinking disabled and 4,096 output tokens.
|
||||
Candidate methods and their workspaces are not dependencies of this plugin.
|
||||
The benchmark runner invokes this Job in a separate application and saves its
|
||||
prompt, provider output and verdict alongside the candidate prediction.
|
||||
|
||||
Sources: [MemBase executable judge](https://github.com/zjunlp/MemBase/blob/a9ba9495d48541d3e68588ff097294cc7a1201bc/membase/inference_utils/prompts.py)
|
||||
and [MobileMem published strict rubric](https://github.com/zjunlp/MobileMem/blob/fa5cd13c9c659699df9caf650987ba2bea8ca085/omni/eval/eval/question_answering_and_judge_prompts.txt).
|
||||
The upstream prompt/parser code retains its MIT notice in `LICENSE-CODE`.
|
||||
22
plugins/mobilemem-judge/pyproject.toml
Normal file
22
plugins/mobilemem-judge/pyproject.toml
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
[project]
|
||||
name = "reme-mobilemem-judge"
|
||||
version = "0.1.0"
|
||||
description = "Official MobileMem-Omni judge protocols as an independent ReMe plugin."
|
||||
readme = "README.md"
|
||||
license = "Apache-2.0"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["reme-ai[as]>=0.4.1.11"]
|
||||
|
||||
[project.entry-points."reme.plugins"]
|
||||
mobilemem-judge = "judge_mobilemem"
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
include = ["judge_mobilemem*"]
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
judge_mobilemem = ["plugin.yaml"]
|
||||
|
||||
[build-system]
|
||||
requires = ["setuptools>=77", "wheel"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
5
plugins/mobilemem-judge/src/judge_mobilemem/__init__.py
Normal file
5
plugins/mobilemem-judge/src/judge_mobilemem/__init__.py
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
"""Independent MobileMem evaluation-only plugin."""
|
||||
|
||||
from .llm_judge import MobileMemJudgeStep
|
||||
|
||||
__all__ = ["MobileMemJudgeStep"]
|
||||
43
plugins/mobilemem-judge/src/judge_mobilemem/llm_judge.py
Normal file
43
plugins/mobilemem-judge/src/judge_mobilemem/llm_judge.py
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
"""Official MobileMem binary judge as an isolated ReMe Job."""
|
||||
|
||||
from dataclasses import asdict
|
||||
|
||||
from agentscope.message import UserMsg
|
||||
|
||||
from reme.steps.base_step import BaseStep
|
||||
from reme.steps.evolve.compressor import CompressorStep
|
||||
|
||||
from .protocol import judge_prompt, parse_judge, protocol_id
|
||||
|
||||
|
||||
class MobileMemJudgeStep(BaseStep):
|
||||
"""Score saved predictions without accessing the candidate memory or tools."""
|
||||
|
||||
async def execute(self):
|
||||
"""Apply the unchanged benchmark prompt and verdict parser."""
|
||||
mode = self.context.get("judge_protocol", "membase_default")
|
||||
prompt = judge_prompt(
|
||||
self.context["question"],
|
||||
self.context["reference"],
|
||||
self.context["prediction"],
|
||||
self.context.get("evidence", ()),
|
||||
judge_protocol=mode,
|
||||
)
|
||||
response = await self.as_llm([UserMsg(name="user", content=prompt)])
|
||||
if hasattr(type(response), "__aiter__"):
|
||||
last = None
|
||||
async for chunk in response:
|
||||
last = chunk
|
||||
response = last
|
||||
raw = await CompressorStep._response_text(response) # pylint: disable=protected-access
|
||||
parsed = parse_judge(raw)
|
||||
self.context.response.success = True
|
||||
self.context.response.answer = str(parsed["llm_judge_score"])
|
||||
self.context.response.metadata.update(
|
||||
**parsed,
|
||||
default_judge_protocol=protocol_id(mode),
|
||||
prompt=prompt,
|
||||
raw_response=raw,
|
||||
usage=asdict(response.usage) if response is not None and response.usage is not None else None,
|
||||
)
|
||||
return self.context.response
|
||||
40
plugins/mobilemem-judge/src/judge_mobilemem/plugin.yaml
Normal file
40
plugins/mobilemem-judge/src/judge_mobilemem/plugin.yaml
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
backends:
|
||||
mobilemem_judge_step: judge_mobilemem.llm_judge:MobileMemJudgeStep
|
||||
|
||||
application_defaults:
|
||||
jobs:
|
||||
answer_judge:
|
||||
backend: base
|
||||
description: "MobileMem-Omni official judge protocol for a saved prediction"
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
question: {type: string}
|
||||
reference: {type: string}
|
||||
prediction: {type: string}
|
||||
evidence: {type: array, default: []}
|
||||
judge_protocol:
|
||||
type: string
|
||||
enum: [membase_default, native_strict]
|
||||
default: membase_default
|
||||
required: [question, reference, prediction]
|
||||
steps:
|
||||
- backend: mobilemem_judge_step
|
||||
as_llm: judge
|
||||
components:
|
||||
as_llm:
|
||||
judge:
|
||||
backend: openai
|
||||
model: ${JUDGE_MODEL_NAME:-qwen3.8-max}
|
||||
stream: false
|
||||
max_retries: 3
|
||||
retry_delay: 60
|
||||
client_kwargs: {timeout: 600, max_retries: 0}
|
||||
credential:
|
||||
api_key: ${LLM_API_KEY:-}
|
||||
base_url: ${LLM_BASE_URL:-}
|
||||
extra_body: {enable_thinking: false}
|
||||
parameters:
|
||||
temperature: 0
|
||||
thinking_enable: false
|
||||
max_tokens: 4096
|
||||
118
plugins/mobilemem-judge/src/judge_mobilemem/protocol.py
Normal file
118
plugins/mobilemem-judge/src/judge_mobilemem/protocol.py
Normal file
|
|
@ -0,0 +1,118 @@
|
|||
"""Unchanged MobileMem/MemBase judge prompts and binary parser.
|
||||
|
||||
Adapted from ZJUNLP's MobileMem and MemBase revisions below.
|
||||
MIT License, Copyright (c) 2026 ZJUNLP; see LICENSE-CODE.
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
from string import Template
|
||||
|
||||
SOURCE_REVISION = "fa5cd13c9c659699df9caf650987ba2bea8ca085"
|
||||
MEMBASE_REVISION = "a9ba9495d48541d3e68588ff097294cc7a1201bc"
|
||||
PROTOCOL_ID = "mobilemem-omni-membase-default-judge-v1"
|
||||
STRICT_PROTOCOL_ID = "mobilemem-omni-final-memory-native-v1"
|
||||
TEMPLATE_NAME = "mobilemem-omni-judge"
|
||||
SOURCE_URL = f"https://github.com/zjunlp/MemBase/blob/{MEMBASE_REVISION}/membase/inference_utils/prompts.py"
|
||||
JUDGE_TEMPLATE = (
|
||||
"Your task is to classify an answer to a question as CORRECT or WRONG. "
|
||||
"You are given a question, a gold answer, a generated answer, and optional "
|
||||
"evidence from the user's prior conversations.\n\n"
|
||||
"GRADING RULES:\n"
|
||||
"1. The generated answer must include every key element in the gold answer.\n"
|
||||
"2. Unsupported or irrelevant content that changes the meaning makes the answer WRONG.\n"
|
||||
"3. Equivalent date formats are acceptable when they refer to the same time.\n"
|
||||
"4. Use the evidence only to verify the answer; do not infer unsupported facts.\n\n"
|
||||
"Question: $question\n"
|
||||
"Gold answer: $golden_answers\n"
|
||||
"Evidence: $evidence\n"
|
||||
"Generated answer: $prediction\n\n"
|
||||
'Return exactly one JSON object: {"label": "CORRECT"} or '
|
||||
'{"label": "WRONG"}.'
|
||||
)
|
||||
STRICT_JUDGE_TEMPLATE = (
|
||||
"Your task is to classify an answer to a question as CORRECT or WRONG. "
|
||||
"You will be provided with the following information:\n"
|
||||
"(1) A question (asked by a user to an AI assistant);\n"
|
||||
"(2) A gold (ground truth) answer;\n"
|
||||
"(3) A generated answer;\n"
|
||||
"(4) A set of evidence sentences relevant to the question, which may be used to verify the generated answer.\n"
|
||||
"The question focuses on information that an AI assistant should recall about the user "
|
||||
"based on prior conversations. "
|
||||
"The gold answer is typically concise and includes all key facts.\n\n"
|
||||
"GRADING RULES (Must Be Strictly Followed)\n"
|
||||
"1. COMPLETE COVERAGE REQUIRED: The generated answer must include every key element present in the gold answer. "
|
||||
"Missing any required fact results in a WRONG classification.\n"
|
||||
"2. NO REDUNDANT CONTENT: Including excessive or irrelevant details not supported by the evidence or gold answer "
|
||||
"results in a WRONG classification.\n"
|
||||
"3. TIME-RELATED QUESTIONS: Different date formats (e.g., May 7th vs. 7 May) are acceptable "
|
||||
"as long as the same date or time period is referenced.\n\n"
|
||||
"INSTRUCTIONS FOR THIS TASK\n"
|
||||
"Use the provided evidence only as supporting context for verification when available. "
|
||||
"Do not infer facts that are not present in the question, gold answer, or evidence.\n"
|
||||
"First, provide a brief (one-sentence) explanation of your reasoning. Then, output the label CORRECT or WRONG.\n"
|
||||
"Do not include both labels in your response, as this will break the evaluation script.\n"
|
||||
"Return the label in JSON format with the key label.\n\n"
|
||||
"QUESTION TO EVALUATE\n"
|
||||
"Question: [Question]\n"
|
||||
"Gold answer: [Gold Answer]\n"
|
||||
"Evidence: [Evidences]\n"
|
||||
"Generated answer: [Generated Answer]"
|
||||
)
|
||||
|
||||
|
||||
def protocol_id(judge_protocol: str) -> str:
|
||||
"""Identify the selected rubric without silently substituting another."""
|
||||
if judge_protocol == "membase_default":
|
||||
return PROTOCOL_ID
|
||||
if judge_protocol == "native_strict":
|
||||
return STRICT_PROTOCOL_ID
|
||||
raise ValueError(f"Unsupported MobileMem judge protocol: {judge_protocol!r}")
|
||||
|
||||
|
||||
def judge_prompt(
|
||||
question: str,
|
||||
reference: str,
|
||||
prediction: str,
|
||||
evidence=(),
|
||||
*,
|
||||
judge_protocol="membase_default",
|
||||
) -> str:
|
||||
"""Preserve native single-pass substitution and untrimmed evidence."""
|
||||
protocol_id(judge_protocol)
|
||||
evidence_text = "\n".join(str(item) for item in evidence) or "Not provided."
|
||||
if judge_protocol == "membase_default":
|
||||
return Template(JUDGE_TEMPLATE).substitute(
|
||||
question=question,
|
||||
golden_answers=reference,
|
||||
evidence=evidence_text,
|
||||
prediction=prediction,
|
||||
)
|
||||
replacements = {
|
||||
"[Question]": question,
|
||||
"[Gold Answer]": reference,
|
||||
"[Evidences]": evidence_text,
|
||||
"[Generated Answer]": prediction,
|
||||
}
|
||||
return re.sub(
|
||||
r"\[Question\]|\[Gold Answer\]|\[Evidences\]|\[Generated Answer\]",
|
||||
lambda match: replacements[match[0]],
|
||||
STRICT_JUDGE_TEMPLATE,
|
||||
)
|
||||
|
||||
|
||||
def parse_judge(text: str) -> dict:
|
||||
"""Preserve MemBase's WRONG-first binary parsing and native default zero."""
|
||||
text = text.strip()
|
||||
try:
|
||||
parsed = json.loads(text)
|
||||
label = str(parsed.get("label", ""))
|
||||
except (json.JSONDecodeError, AttributeError):
|
||||
label = text
|
||||
normalised = label.strip().upper()
|
||||
score = 0.0 if "WRONG" in normalised else float("CORRECT" in normalised)
|
||||
return {
|
||||
"llm_judge_score": score,
|
||||
"label": "CORRECT" if score else "WRONG",
|
||||
"parse_default_zero": "WRONG" not in normalised and "CORRECT" not in normalised,
|
||||
}
|
||||
17
plugins/mobilemem/README.md
Normal file
17
plugins/mobilemem/README.md
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
# MobileMem plugin
|
||||
|
||||
Install this plugin from the repository:
|
||||
|
||||
```bash
|
||||
pip install -e plugins/mobilemem
|
||||
```
|
||||
|
||||
Enable the `mobilemem` ReMe plugin for the MobileMem-Omni benchmark. Its Steps use native ReMe Auto Memory,
|
||||
source-line citations, BM25 retrieval, workspace containment, and an AgentScope bridge for the native image
|
||||
reader. The plugin installs independently of MemLens. Evaluation uses the separate
|
||||
[`mobilemem-judge`](../mobilemem-judge/README.md) plugin.
|
||||
|
||||
The runner supplies only original dialogue messages and image attachments. Memory extraction retains dates,
|
||||
updates, personal relationships, app details and visual evidence; each question uses a fresh answer Agent. Source
|
||||
session reading and `read_image` remain scoped to that user's workspace. Supplied captions and benchmark gold
|
||||
annotations must never be passed to either Job.
|
||||
25
plugins/mobilemem/pyproject.toml
Normal file
25
plugins/mobilemem/pyproject.toml
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
[project]
|
||||
name = "reme-mobilemem"
|
||||
version = "0.1.0"
|
||||
description = "MobileMem-Omni native multimodal memory benchmark plugin for ReMe."
|
||||
readme = "README.md"
|
||||
license = "Apache-2.0"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = ["reme-ai[as]>=0.4.1.11"]
|
||||
|
||||
[project.entry-points."reme.plugins"]
|
||||
mobilemem = "reme_mobilemem"
|
||||
|
||||
[project.entry-points."reme.configs"]
|
||||
mobilemem-benchmark = "reme_mobilemem:BENCHMARK_CONFIG"
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
include = ["reme_mobilemem*"]
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
reme_mobilemem = ["plugin.yaml", "*.yaml"]
|
||||
|
||||
[build-system]
|
||||
requires = ["setuptools>=77", "wheel"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
5
plugins/mobilemem/src/reme_mobilemem/__init__.py
Normal file
5
plugins/mobilemem/src/reme_mobilemem/__init__.py
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
"""MobileMem-Omni benchmark Jobs for ReMe."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
BENCHMARK_CONFIG = Path(__file__).with_name("benchmark.yaml")
|
||||
137
plugins/mobilemem/src/reme_mobilemem/agentic_answer.py
Normal file
137
plugins/mobilemem/src/reme_mobilemem/agentic_answer.py
Normal file
|
|
@ -0,0 +1,137 @@
|
|||
"""Agentic BM25 answer with scene-local, optional multimodal source access."""
|
||||
|
||||
from pathlib import Path
|
||||
from uuid import uuid4
|
||||
|
||||
from agentscope.message import Base64Source, DataBlock, TextBlock, ToolResultState, UserMsg
|
||||
from agentscope.tool import FunctionTool, ToolChunk, Toolkit
|
||||
|
||||
from reme.components.agent_wrapper.as_agent_wrapper import AsAgentWrapper
|
||||
from reme.steps.base_step import BaseStep
|
||||
from reme.steps.file_io._path import _check_path_permission, resolve_path
|
||||
|
||||
|
||||
def evidence_paths(workspace: Path, daily_dir: str, digest_dir: str, session_dir: str, sources: bool) -> list[str]:
|
||||
"""List exact allowed evidence files, never agent state, settings, or another workspace."""
|
||||
roots = [(daily_dir, "*.md"), (digest_dir, "*.md")]
|
||||
if sources:
|
||||
roots.append((f"{session_dir}/dialog", "*.jsonl"))
|
||||
workspace = workspace.resolve()
|
||||
paths = []
|
||||
for directory, pattern in roots:
|
||||
root = (workspace / directory).resolve()
|
||||
if not root.is_relative_to(workspace):
|
||||
raise ValueError("Evidence directory must stay inside this scene workspace")
|
||||
for path in sorted(root.rglob(pattern)):
|
||||
if path.is_file() and path.resolve().is_relative_to(root):
|
||||
paths.append(path.relative_to(workspace).as_posix())
|
||||
return paths
|
||||
|
||||
|
||||
def image_reader(step: BaseStep) -> FunctionTool:
|
||||
"""Bridge the native read_image Job response to an AgentScope image result."""
|
||||
workspace = step.workspace_path.resolve()
|
||||
image_directory = f"{step.config_value('session_dir')}/images"
|
||||
|
||||
async def read_image(path: str) -> ToolChunk:
|
||||
"""View an image using its session/images path cited in a memory note."""
|
||||
target, error = resolve_path(workspace, path)
|
||||
if error or target is None or not _check_path_permission(workspace, target, [image_directory]):
|
||||
return ToolChunk(
|
||||
content=[TextBlock(text=error or "Image path must stay inside this scene's session image directory.")],
|
||||
state=ToolResultState.ERROR,
|
||||
)
|
||||
relative = target.relative_to(workspace).as_posix()
|
||||
response = await step.run_job("read_image", path=relative)
|
||||
metadata = response.metadata
|
||||
media_type = metadata.get("mime", "")
|
||||
if (
|
||||
not response.success
|
||||
or metadata.get("oversized")
|
||||
or metadata.get("non_image_warning")
|
||||
or not media_type.startswith("image/")
|
||||
):
|
||||
notice = str(response.answer) if not response.success or metadata.get("oversized") else "Not an image file."
|
||||
return ToolChunk(content=[TextBlock(text=notice)], state=ToolResultState.ERROR)
|
||||
image = DataBlock(source=Base64Source(media_type=media_type, data=response.answer))
|
||||
return ToolChunk(content=[TextBlock(text=f"Image source: {relative}"), image], state=ToolResultState.SUCCESS)
|
||||
|
||||
return FunctionTool(func=read_image, name="read_image", is_concurrency_safe=True, is_read_only=True)
|
||||
|
||||
|
||||
class MobileMemAgenticAnswerStep(BaseStep):
|
||||
"""Run a fresh Agent for each question, with the official answer prompt unchanged."""
|
||||
|
||||
async def execute(self):
|
||||
"""Answer through a fresh, scene-scoped Agent and its allowed tools."""
|
||||
self.context.data.setdefault("max_iters", 30)
|
||||
wrapper = self.agent_wrapper
|
||||
if not isinstance(wrapper, AsAgentWrapper):
|
||||
raise NotImplementedError("MobileMem agentic_answer requires the AgentScope wrapper")
|
||||
query = self.context.get("query", "")
|
||||
if not query:
|
||||
raise ValueError("query must contain the benchmark answer prompt")
|
||||
max_iters = self.context.get("max_iters", 30)
|
||||
if isinstance(max_iters, bool) or not isinstance(max_iters, int) or max_iters < 1:
|
||||
raise ValueError("max_iters must be a positive integer")
|
||||
sources = bool(self.context.get("include_source_sessions", True))
|
||||
images = bool(self.context.get("include_images", False))
|
||||
paths = evidence_paths(
|
||||
self.workspace_path,
|
||||
self.config_value("daily_dir"),
|
||||
self.config_value("digest_dir"),
|
||||
self.config_value("session_dir"),
|
||||
sources,
|
||||
)
|
||||
tool_context_id = f"mobilemem_answer_{uuid4().hex}"
|
||||
tools = []
|
||||
if images:
|
||||
tools.append(image_reader(self))
|
||||
injected = {
|
||||
"_allowed_paths": paths,
|
||||
"read_step_format_session": True,
|
||||
"search_filter": {"paths": paths},
|
||||
"vector_weight": 0.0,
|
||||
}
|
||||
# Standard ReMe job tools keep their normal file handling and lifecycle.
|
||||
tools.extend(
|
||||
wrapper._make_tool(job, tool_context_id, injected) # pylint: disable=protected-access
|
||||
for job in wrapper._resolve_job_tools(["search", "read"]) # pylint: disable=protected-access
|
||||
)
|
||||
blocks = [TextBlock(text=query)]
|
||||
blocks.extend(DataBlock.model_validate(block) for block in self.context.get("query_images", []))
|
||||
if any(not block.source.media_type.startswith("image/") for block in blocks[1:]):
|
||||
raise ValueError("query_images accepts image DataBlocks only")
|
||||
prompt = self.get_prompt("system_prompt")
|
||||
if query_time := self.context.get("query_time"):
|
||||
prompt += f"\nQuestion time: {query_time}"
|
||||
if images:
|
||||
prompt += (
|
||||
"\nUse read_image(path=...) for the session/images paths cited in memory, including source_images."
|
||||
)
|
||||
else:
|
||||
prompt += "\nHistory images cannot be reopened in this setting."
|
||||
try:
|
||||
result = await wrapper.reply(
|
||||
UserMsg(name="user", content=blocks),
|
||||
system_prompt=prompt,
|
||||
toolkit=Toolkit(tools=tools),
|
||||
job_tools=[],
|
||||
builtin_tools=False,
|
||||
session_id=str(uuid4()),
|
||||
resume="",
|
||||
react_config={"max_iters": max_iters},
|
||||
)
|
||||
self.context.response.answer = (result.get("result") or "").strip()
|
||||
self.context.response.metadata.update(
|
||||
usage=result.get("usage"),
|
||||
agentscope_usage=(result.get("last_message") or {}).get("usage"),
|
||||
agent_session_id=result.get("session_id"),
|
||||
include_source_sessions=sources,
|
||||
include_images=images,
|
||||
max_iters=max_iters,
|
||||
)
|
||||
finally:
|
||||
if self.app_context is not None:
|
||||
self.app_context.metadata.get("tool_contexts", {}).pop(tool_context_id, None)
|
||||
return self.context.response
|
||||
8
plugins/mobilemem/src/reme_mobilemem/agentic_answer.yaml
Normal file
8
plugins/mobilemem/src/reme_mobilemem/agentic_answer.yaml
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
system_prompt: |
|
||||
Answer the supplied MobileMem question from this user's saved memories and original source conversations.
|
||||
Search for the relevant people, events, dates and distinctive details, then read the source files or exact line
|
||||
ranges when additional context is needed. Distinguish changes over time and similar events. When read_image is
|
||||
available, reopen cited session/images paths to verify visual details rather than guessing them from an ID.
|
||||
Use another evidence search when the first result is incomplete. Follow the supplied benchmark answer rules.
|
||||
Include all facts needed to answer the question, without unsupported or irrelevant details. Do not include the
|
||||
retrieval process or tool transcript in the final answer.
|
||||
34
plugins/mobilemem/src/reme_mobilemem/auto_memory.py
Normal file
34
plugins/mobilemem/src/reme_mobilemem/auto_memory.py
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
"""MobileMem memory prompts with optional, exact source-line references."""
|
||||
|
||||
from agentscope.message import Msg
|
||||
|
||||
from reme.steps.evolve.auto_memory import AutoMemoryStep
|
||||
|
||||
|
||||
class MobileMemAutoMemoryStep(AutoMemoryStep):
|
||||
"""Keep core persistence and multimodal input, adding benchmark instructions only."""
|
||||
|
||||
def _format_history(self, messages: list[Msg]) -> str:
|
||||
if not self.context.get("include_source_sessions", True):
|
||||
return "Source-session access is disabled; record self-contained facts.\n\n" + super()._format_history(
|
||||
messages,
|
||||
)
|
||||
|
||||
# Core execute has saved the session already. Read physical line numbers
|
||||
# rather than assuming input order equals timestamp-sorted JSONL order.
|
||||
session_id = self.context.get("session_id", "")
|
||||
source = self._session_source_path(session_id)
|
||||
# File iteration keeps Unicode line separators inside JSON strings intact.
|
||||
with self._session_path(session_id).open(encoding="utf-8") as stream:
|
||||
lines = {Msg.model_validate_json(line).id: number for number, line in enumerate(stream, 1) if line.strip()}
|
||||
turns = [
|
||||
f"[L{lines[msg.id]} | {msg.name or msg.role} @ {msg.created_at}]\n{msg.get_text_content() or ''}"
|
||||
for msg in messages
|
||||
]
|
||||
return (
|
||||
f"Source file: {source}\n"
|
||||
f"Cite useful evidence with the full source path, for example [[{source}#L1-L2]], "
|
||||
"replacing the line range with the actual physical lines below. Never use bare links such as [[L5]]. "
|
||||
"Preserve original Image IDs alongside their facts, and retain the supplied Image source paths.\n\n"
|
||||
+ "\n\n".join(turns)
|
||||
)
|
||||
17
plugins/mobilemem/src/reme_mobilemem/auto_memory.yaml
Normal file
17
plugins/mobilemem/src/reme_mobilemem/auto_memory.yaml
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
system_prompt: |
|
||||
Maintain the user's long-term mobile memory from this original dated conversation. Save concrete personal facts,
|
||||
names, relationships, dates, quantities, locations, plans, decisions and preferences. Distinguish separate events
|
||||
across the year and preserve earlier dated facts when newer information updates them. Keep whose experience
|
||||
each fact describes explicit. Use the supplied session timestamp for relative dates only when justified.
|
||||
|
||||
When original images are present, understand them jointly with surrounding text. Retain distinctive visual
|
||||
details, visible app text, amounts, dates, people, objects and activities. Preserve each supplied Image ID and
|
||||
image source path with its facts so the original can be reopened. An identifier is not evidence of its contents.
|
||||
Do not infer unavailable image details or introduce outside information.
|
||||
|
||||
Cite useful physical source-line ranges with the full supplied source path when source access is enabled.
|
||||
Otherwise retain self-contained facts. Follow the normal daily-note contract with a stable filename stem,
|
||||
informative description and factual body. Never set status. Use structured file tool calls to save the memory.
|
||||
|
||||
Create at most one new note for this session. After a successful daily_write, finish with the requested
|
||||
one-sentence summary. Further writes are needed only to correct a tool error or a concrete omission.
|
||||
32
plugins/mobilemem/src/reme_mobilemem/benchmark.yaml
Normal file
32
plugins/mobilemem/src/reme_mobilemem/benchmark.yaml
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
extends: benchmark
|
||||
plugins: [mobilemem]
|
||||
components:
|
||||
file_store:
|
||||
default:
|
||||
embedding_store: null
|
||||
as_llm:
|
||||
default: &candidate_model
|
||||
backend: openai
|
||||
model: ${LLM_MODEL_NAME:-qwen3.8-max}
|
||||
stream: false
|
||||
context_size: 1000000
|
||||
max_retries: 3
|
||||
retry_delay: 60
|
||||
credential:
|
||||
api_key: ${LLM_API_KEY:-}
|
||||
base_url: ${LLM_BASE_URL:-}
|
||||
extra_body: {enable_thinking: true}
|
||||
parameters:
|
||||
temperature: 0
|
||||
thinking_enable: true
|
||||
max_tokens: 65536
|
||||
client_kwargs: {timeout: 600, max_retries: 0}
|
||||
bench: *candidate_model
|
||||
compressor: *candidate_model
|
||||
agent_wrapper:
|
||||
default:
|
||||
context_config: {max_image_num: 4096}
|
||||
model_config: {max_retries: 0}
|
||||
bench:
|
||||
context_config: {max_image_num: 4096}
|
||||
model_config: {max_retries: 0}
|
||||
70
plugins/mobilemem/src/reme_mobilemem/plugin.yaml
Normal file
70
plugins/mobilemem/src/reme_mobilemem/plugin.yaml
Normal file
|
|
@ -0,0 +1,70 @@
|
|||
backends:
|
||||
mobilemem_auto_memory_step: reme_mobilemem.auto_memory:MobileMemAutoMemoryStep
|
||||
mobilemem_agentic_answer_step: reme_mobilemem.agentic_answer:MobileMemAgenticAnswerStep
|
||||
|
||||
application_defaults:
|
||||
jobs:
|
||||
read_image:
|
||||
backend: base
|
||||
description: "Read an original saved session image through ReMe's native image Job."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
path: {type: string, description: "Workspace-relative image attachment path."}
|
||||
required: [path]
|
||||
steps:
|
||||
- backend: read_image_step
|
||||
max_bytes: 52428800
|
||||
auto_memory:
|
||||
backend: base
|
||||
description: "Record one original MobileMem-Omni session using native Auto Memory."
|
||||
include_source_sessions: true
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
messages: {type: array, items: {type: object}}
|
||||
session_id: {type: string}
|
||||
date: {type: string, default: ""}
|
||||
include_images: {type: boolean, default: true}
|
||||
include_source_sessions: {type: boolean, default: true}
|
||||
memory_hint: {type: string, default: ""}
|
||||
required: [messages, session_id]
|
||||
steps:
|
||||
- backend: mobilemem_auto_memory_step
|
||||
search:
|
||||
backend: base
|
||||
description: "Search this user's allowed memory and source text using BM25."
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
query: {type: string}
|
||||
limit: {type: integer, description: "Number of search results."}
|
||||
start_date: {type: string}
|
||||
end_date: {type: string}
|
||||
required: [query]
|
||||
steps:
|
||||
- backend: search_step
|
||||
vector_weight: 0.0
|
||||
expand_links: false
|
||||
agentic_answer:
|
||||
backend: base
|
||||
description: "Answer a MobileMem question through a fresh evidence retrieval Agent."
|
||||
include_source_sessions: true
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
query: {type: string, description: "Official benchmark answer prompt including the query."}
|
||||
query_time: {type: string, default: ""}
|
||||
query_images: {type: array, items: {type: object}, default: []}
|
||||
scene_id: {type: string}
|
||||
max_iters: {type: integer, minimum: 1, default: 30}
|
||||
include_images: {type: boolean, default: true}
|
||||
include_source_sessions: {type: boolean, default: true}
|
||||
required: [query, scene_id]
|
||||
steps:
|
||||
- backend: mobilemem_agentic_answer_step
|
||||
agent_wrapper: bench
|
||||
components:
|
||||
file_store:
|
||||
default:
|
||||
embedding_store: null
|
||||
373
tests/unit/test_mobilemem_benchmark.py
Normal file
373
tests/unit/test_mobilemem_benchmark.py
Normal file
|
|
@ -0,0 +1,373 @@
|
|||
"""Focused offline benchmark contracts; no external services or experiment writes."""
|
||||
|
||||
# pylint: disable=missing-function-docstring,protected-access
|
||||
import base64
|
||||
from importlib import import_module
|
||||
from importlib.metadata import EntryPoint
|
||||
from io import BytesIO
|
||||
import json
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
from agentscope.formatter import OpenAIChatFormatter
|
||||
from agentscope.message import AssistantMsg, Base64Source, DataBlock, Msg, TextBlock, ToolResultBlock, ToolResultState
|
||||
from PIL import Image
|
||||
import pytest
|
||||
|
||||
from reme.components import R
|
||||
from reme.components.agent_wrapper.as_agent_wrapper import AsAgentWrapper
|
||||
from reme.components.component_registry import create_application_registry
|
||||
from reme.components.file_store import LocalFileStore
|
||||
from reme.plugin import PluginManager, _load_plugin
|
||||
from reme.schema import ApplicationConfig, Response
|
||||
from reme.steps.evolve.auto_memory import _sanitize_msg_for_save
|
||||
from reme.steps.file_io.read_image import ReadImageStep
|
||||
|
||||
from benchmark.mobilemem import run as mobilemem
|
||||
|
||||
|
||||
@pytest.fixture(name="mobile_scene")
|
||||
def fixture_mobile_scene(tmp_path):
|
||||
folder = tmp_path / "image/uid0/camera_photos"
|
||||
folder.mkdir(parents=True)
|
||||
Image.new("RGB", (3, 2), (12, 34, 56)).save(folder / "photo.png")
|
||||
sessions = [
|
||||
{
|
||||
"session_id": f"0_{index}",
|
||||
"event_start_time": f"2025-0{index}-01 09:00:00",
|
||||
"event_end_time": f"2025-0{index}-01 10:00:00",
|
||||
"dialogue": [
|
||||
{"turn": 1, "role": "user", "content_type": "text", "content": f"Original turn {index}"},
|
||||
{"turn": 2, "role": "user", "content_type": "image", "image_inline": "image/uid0/event/photo.png"},
|
||||
{"turn": 3, "role": "assistant", "content_type": "text", "content": "Original reply"},
|
||||
],
|
||||
"dialogue_summary": "HIDDEN_SUMMARY",
|
||||
"memory_points": [{"memory_content": "HIDDEN_MEMORY"}],
|
||||
}
|
||||
for index in (2, 1)
|
||||
]
|
||||
question = {
|
||||
"question_id": "0_q_1",
|
||||
"question": "What happened?",
|
||||
"answer": "HIDDEN_GOLD",
|
||||
"question_format": "open_ended",
|
||||
"question_type": "visual_reasoning",
|
||||
"source_session_ids": ["0_1"],
|
||||
"image_refs": ["HIDDEN_IMAGE"],
|
||||
"evidence": [{"explanation": "HIDDEN_EVIDENCE"}],
|
||||
}
|
||||
(tmp_path / "data.jsonl").write_text(json.dumps({"uuid": 0, "language": "en", "sessions": sessions}) + "\n")
|
||||
(tmp_path / "questions.jsonl").write_text(json.dumps({"uuid": 0, "questions": [question]}) + "\n")
|
||||
return tmp_path, mobilemem.load_mobilemem(tmp_path)[0]
|
||||
|
||||
|
||||
def test_mobilemem_complete_history_ids_and_no_gold(mobile_scene):
|
||||
root, scene = mobile_scene
|
||||
assert [session["session_id"] for session in scene["sessions"]] == ["0_1", "0_2"]
|
||||
assert scene["questions"][0]["question_type"] == "visual_reasoning"
|
||||
converted = mobilemem.session_messages(scene["sessions"][0], scene["image_root"])
|
||||
assert "HIDDEN" not in json.dumps(converted)
|
||||
original = root / "image/uid0/camera_photos/photo.png"
|
||||
before = original.read_bytes()
|
||||
assert converted[1]["content"][1]["text"] == "[Image ID: image/uid0/event/photo.png]"
|
||||
assert converted[1]["content"][2]["type"] == "data"
|
||||
assert base64.b64decode(converted[1]["content"][2]["source"]["data"]) == before
|
||||
assert original.read_bytes() == before
|
||||
with pytest.raises(ValueError, match="Invalid MobileMem image path"):
|
||||
mobilemem.resolve_image(root / "image", "../outside.png", {})
|
||||
original.unlink()
|
||||
with pytest.raises(ValueError, match="Missing or ambiguous MobileMem image"):
|
||||
mobilemem.session_messages(scene["sessions"][0], scene["image_root"])
|
||||
|
||||
|
||||
def test_official_query_and_overlap_metrics():
|
||||
assert mobilemem.query_prompt("QUESTION", "CONTEXT") == (
|
||||
"Question:\nQUESTION\nPlease answer the question based on the following memories:\nCONTEXT\n"
|
||||
+ mobilemem.ANSWER_RULES
|
||||
)
|
||||
score = mobilemem.score_answer("北京 北京", "北京")
|
||||
assert score["f1"] == pytest.approx(2 / 3)
|
||||
assert score["bleu_1"] == pytest.approx(0.36787944117144233)
|
||||
|
||||
|
||||
@pytest.fixture(name="candidate")
|
||||
def fixture_candidate(tmp_path, monkeypatch):
|
||||
name = "mobilemem"
|
||||
monkeypatch.syspath_prepend(str(Path(__file__).resolve().parents[2] / "plugins" / name / "src"))
|
||||
monkeypatch.chdir(tmp_path)
|
||||
memory = import_module(f"reme_{name}.auto_memory")
|
||||
answer = import_module(f"reme_{name}.agentic_answer")
|
||||
prefix = "MobileMem"
|
||||
app = SimpleNamespace(
|
||||
registry=R,
|
||||
metadata={},
|
||||
app_config=ApplicationConfig(workspace_dir=str(tmp_path)),
|
||||
jobs={},
|
||||
components={},
|
||||
)
|
||||
wrapper = AsAgentWrapper(backend="agentscope", as_llm="", app_context=app)
|
||||
wrapper.reply = AsyncMock(return_value={"result": "answer", "session_id": "new", "usage": {"input_tokens": 7}})
|
||||
store = LocalFileStore(embedding_store="", app_context=app)
|
||||
return name, memory, answer, prefix, app, wrapper, store, tmp_path
|
||||
|
||||
|
||||
def test_candidate_registration_is_application_local(candidate):
|
||||
name, memory, answer, prefix, *_ = candidate
|
||||
plugin = _load_plugin(name, EntryPoint(name=name, value=f"reme_{name}", group="reme.plugins"))
|
||||
registry = create_application_registry()
|
||||
PluginManager([plugin]).register(registry)
|
||||
assert registry.get("step", f"{name}_auto_memory_step") is getattr(memory, f"{prefix}AutoMemoryStep")
|
||||
assert registry.get("step", f"{name}_agentic_answer_step") is getattr(answer, f"{prefix}AgenticAnswerStep")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_candidate_and_judge_start_with_native_settings(tmp_path, monkeypatch):
|
||||
from reme import Application
|
||||
|
||||
root = Path(__file__).resolve().parents[2] / "plugins"
|
||||
for package in ("mobilemem", "mobilemem-judge"):
|
||||
monkeypatch.syspath_prepend(str(root / package / "src"))
|
||||
entries = [
|
||||
EntryPoint(name="mobilemem", value="reme_mobilemem", group="reme.plugins"),
|
||||
EntryPoint(name="mobilemem-judge", value="judge_mobilemem", group="reme.plugins"),
|
||||
EntryPoint(name="mobilemem-benchmark", value="reme_mobilemem:BENCHMARK_CONFIG", group="reme.configs"),
|
||||
]
|
||||
|
||||
def installed(group, name):
|
||||
return [entry for entry in entries if entry.group == group and entry.name == name]
|
||||
|
||||
monkeypatch.setattr("reme.plugin.find_entry_points", installed)
|
||||
monkeypatch.setattr("reme.config.config_parser.find_entry_points", installed)
|
||||
monkeypatch.setenv("LLM_API_KEY", "offline")
|
||||
monkeypatch.setenv("LLM_BASE_URL", "https://provider.invalid/v1")
|
||||
candidate_app = Application(**mobilemem.candidate_config(tmp_path / "candidate", "candidate-model"))
|
||||
judge_app = Application(**mobilemem.judge_config(tmp_path / "judge"))
|
||||
try:
|
||||
await candidate_app.start()
|
||||
await judge_app.start()
|
||||
assert set(judge_app.context.jobs) == {"answer_judge"}
|
||||
assert set(judge_app.context.components) == {"as_llm"}
|
||||
models = candidate_app.context.components["as_llm"]
|
||||
for component in models.values():
|
||||
assert component.kwargs["model"] == "candidate-model"
|
||||
assert component.kwargs["stream"] is False and component.kwargs["context_size"] == 1000000
|
||||
assert component.kwargs["parameters"] == {"max_tokens": 65536, "thinking_enable": True, "temperature": 0}
|
||||
assert component.kwargs["max_retries"] == 3 and component.kwargs["retry_delay"] == 60
|
||||
assert component.kwargs["client_kwargs"] == {"timeout": 600, "max_retries": 0}
|
||||
assert candidate_app.config.components["file_store"]["default"].model_dump()["embedding_store"] is None
|
||||
assert candidate_app.config.jobs["index_update"].model_dump()["watch_dirs"] == [
|
||||
"daily_dir",
|
||||
"digest_dir",
|
||||
"session_dir/dialog",
|
||||
]
|
||||
finally:
|
||||
await judge_app.close()
|
||||
await candidate_app.close()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("sources", [True, False])
|
||||
async def test_memory_saves_originals_and_native_image_links(candidate, sources):
|
||||
_, memory, _, prefix, app, wrapper, store, workspace = candidate
|
||||
step = getattr(memory, f"{prefix}AutoMemoryStep")(app_context=app, agent_wrapper=wrapper, file_store=store)
|
||||
step._list_session_note = AsyncMock(return_value=None)
|
||||
picture = BytesIO()
|
||||
Image.new("RGB", (1, 1), (12, 34, 56)).save(picture, format="PNG")
|
||||
image_bytes = picture.getvalue()
|
||||
image_data = base64.b64encode(image_bytes).decode()
|
||||
late = Msg.model_validate(
|
||||
{
|
||||
"id": "late",
|
||||
"name": "Alice",
|
||||
"role": "user",
|
||||
"created_at": "2026-01-02T10:00:00",
|
||||
"content": [
|
||||
TextBlock(text="Image ID: image-7"),
|
||||
DataBlock(source=Base64Source(media_type="image/png", data=image_data)),
|
||||
TextBlock(text="Before.\u2028\u2028After.\u2029End.\u0085Done."),
|
||||
],
|
||||
},
|
||||
)
|
||||
early = late.model_copy(update={"id": "early", "created_at": "2026-01-01T10:00:00"}, deep=True)
|
||||
before = late.model_dump()
|
||||
await step(messages=[late, early], session_id="one", include_images=True, include_source_sessions=sources)
|
||||
saved = workspace / "session/dialog/one.jsonl"
|
||||
assert saved.read_text() == "".join(_sanitize_msg_for_save(msg).model_dump_json() + "\n" for msg in [early, late])
|
||||
inputs = wrapper.reply.call_args.args[0]
|
||||
assert [block.type for block in inputs.content] == ["text", "data", "text", "data", "text"]
|
||||
assert [block.source.data for block in inputs.content if isinstance(block, DataBlock)] == [image_data] * 2
|
||||
text = inputs.get_text_content()
|
||||
assert "Image ID: image-7" in text and "Before.\u2028\u2028After.\u2029End.\u0085Done." in text
|
||||
assert ("[L2 | Alice" in text) is sources
|
||||
files = sorted((workspace / "session/images/one").glob("*"))
|
||||
assert len(files) == 2 and all(path.read_bytes() == image_bytes for path in files)
|
||||
assert all(f"[[{path.relative_to(workspace).as_posix()}]]" in text for path in files)
|
||||
assert late.model_dump() == before
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_read_image_returns_real_formatter_image_and_rejects_escape(candidate):
|
||||
_, _, answer, prefix, app, wrapper, store, workspace = candidate
|
||||
path = workspace / "session/images/one/photo.png"
|
||||
path.parent.mkdir(parents=True)
|
||||
Image.new("RGB", (2, 2), (12, 34, 56)).save(path)
|
||||
|
||||
async def native_read(**kwargs):
|
||||
step = ReadImageStep(app_context=app)
|
||||
await step(**kwargs)
|
||||
return step.context.response
|
||||
|
||||
app.jobs["read_image"] = AsyncMock(side_effect=native_read)
|
||||
step = getattr(answer, f"{prefix}AgenticAnswerStep")(app_context=app, agent_wrapper=wrapper, file_store=store)
|
||||
tool = answer.image_reader(step)
|
||||
result = await tool.call(path="session/images/one/photo.png")
|
||||
assert result.state == ToolResultState.SUCCESS and isinstance(result.content[1], DataBlock)
|
||||
assert base64.b64decode(result.content[1].source.data) == path.read_bytes()
|
||||
formatted = await OpenAIChatFormatter().format(
|
||||
[
|
||||
AssistantMsg(name="tool", content=[ToolResultBlock(id="call", name="read_image", output=result.content)]),
|
||||
],
|
||||
)
|
||||
assert any(
|
||||
block["type"] == "image_url"
|
||||
for msg in formatted
|
||||
if isinstance(msg.get("content"), list)
|
||||
for block in msg["content"]
|
||||
)
|
||||
assert (await tool.call(path="../other-scene.png")).state == ToolResultState.ERROR
|
||||
assert (await tool.call(path="mem_session/private.png")).state == ToolResultState.ERROR
|
||||
app.jobs["read_image"].assert_awaited_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize("sources", [True, False])
|
||||
async def test_answer_has_fresh_state_bm25_scope_and_no_gold(candidate, sources):
|
||||
_, _, answer, prefix, app, wrapper, store, workspace = candidate
|
||||
for relative in ["daily/note.md", "digest/note.md", "session/dialog/one.jsonl", "mem_session/private.md"]:
|
||||
path = workspace / relative
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text("evidence")
|
||||
assert answer.evidence_paths(workspace, "daily", "digest", "session", False) == ["daily/note.md", "digest/note.md"]
|
||||
calls = []
|
||||
|
||||
async def search(**kwargs):
|
||||
calls.append(kwargs)
|
||||
return Response(answer="evidence")
|
||||
|
||||
job = AsyncMock(side_effect=search)
|
||||
job.name, job.description, job.parameters = (
|
||||
"search",
|
||||
"search",
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {"query": {"type": "string"}},
|
||||
},
|
||||
)
|
||||
read = AsyncMock(return_value=Response(answer="evidence"))
|
||||
read.name, read.description, read.parameters = "read", "read", {"type": "object", "properties": {}}
|
||||
app.jobs = {"search": job, "read": read}
|
||||
step = getattr(answer, f"{prefix}AgenticAnswerStep")(app_context=app, agent_wrapper=wrapper, file_store=store)
|
||||
sessions, contexts = [], []
|
||||
for _ in range(2):
|
||||
response = await step(query="Official question", include_source_sessions=sources)
|
||||
options = wrapper.reply.call_args.kwargs
|
||||
assert wrapper.reply.call_args.args[0].get_text_content() == "Official question"
|
||||
assert options["resume"] == "" and options["builtin_tools"] is False and options["job_tools"] == []
|
||||
assert options["react_config"] == {"max_iters": 30}
|
||||
sessions.append(options["session_id"])
|
||||
await (await options["toolkit"].get_tool("search")).call(query="evidence")
|
||||
assert calls[-1]["vector_weight"] == 0.0
|
||||
assert ("session/dialog/one.jsonl" in calls[-1]["search_filter"]["paths"]) is sources
|
||||
assert "mem_session/private.md" not in calls[-1]["_allowed_paths"]
|
||||
contexts.append(calls[-1]["tool_context_id"])
|
||||
assert response.metadata["usage"] == {"input_tokens": 7}
|
||||
assert len(set(sessions)) == len(set(contexts)) == 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_image_only_pipeline_keeps_all_sessions_and_gold_is_judge_only(mobile_scene, tmp_path, monkeypatch):
|
||||
from benchmark.mobilemem.run import normalized_usage, observe_completion
|
||||
|
||||
_, scene = mobile_scene
|
||||
calls, apps = [], []
|
||||
|
||||
class Candidate:
|
||||
"""Offline Application stand-in preserving the native Job call boundary."""
|
||||
|
||||
def __init__(self, **config):
|
||||
self.config = config
|
||||
self.closed = False
|
||||
self.context = SimpleNamespace(
|
||||
components={
|
||||
"agent_wrapper": {name: SimpleNamespace(session_path=tmp_path) for name in ("default", "bench")},
|
||||
},
|
||||
)
|
||||
apps.append(self)
|
||||
|
||||
async def start(self):
|
||||
pass
|
||||
|
||||
async def close(self):
|
||||
self.closed = True
|
||||
|
||||
async def run_job(self, name, **kwargs):
|
||||
calls.append((name, kwargs))
|
||||
return Response(answer="Retained answer", metadata={"agent_session_id": "new-agent"})
|
||||
|
||||
judge = SimpleNamespace(
|
||||
run_job=AsyncMock(
|
||||
return_value=Response(
|
||||
metadata={"llm_judge_score": 1, "parse_default_zero": False},
|
||||
),
|
||||
),
|
||||
)
|
||||
monkeypatch.setattr(mobilemem, "Application", Candidate)
|
||||
|
||||
def configured(workspace, model):
|
||||
return {"workspace": workspace, "model": model}
|
||||
|
||||
monkeypatch.setattr(mobilemem, "candidate_config", configured)
|
||||
monkeypatch.setattr(mobilemem, "observe_usage", lambda _: None)
|
||||
monkeypatch.setattr(mobilemem, "observe_completion", lambda _: None)
|
||||
rows = await mobilemem.evaluate_user(
|
||||
scene,
|
||||
tmp_path / "new-user",
|
||||
"candidate",
|
||||
judge,
|
||||
__import__("asyncio").Semaphore(2),
|
||||
)
|
||||
assert [name for name, _ in calls] == ["auto_memory", "auto_memory", "index_update", "agentic_answer"]
|
||||
assert [kw["session_id"] for name, kw in calls if name == "auto_memory"] == ["0_1", "0_2"]
|
||||
for name, kwargs in calls:
|
||||
assert "HIDDEN" not in json.dumps(kwargs)
|
||||
if name in {"auto_memory", "agentic_answer"}:
|
||||
assert kwargs["include_images"] is kwargs["include_source_sessions"] is True
|
||||
answer = calls[-1][1]
|
||||
assert answer["max_iters"] == 30
|
||||
assert answer["query"] == mobilemem.query_prompt(
|
||||
scene["questions"][0]["question"],
|
||||
"Use the available evidence tools.",
|
||||
)
|
||||
assert "category" not in answer and "question_type" not in answer
|
||||
judge_inputs = judge.run_job.call_args.kwargs
|
||||
assert judge_inputs["reference"] == "HIDDEN_GOLD"
|
||||
assert judge_inputs["evidence"] == ["HIDDEN_EVIDENCE"]
|
||||
assert rows[0]["metrics"]["llm_judge"] == 1 and apps[0].closed
|
||||
assert (tmp_path / "new-user/questions/0000/result.json").is_file()
|
||||
assert normalized_usage({"prompt_tokens": 100, "completion_tokens": 20})["cached_input_tokens"] is None
|
||||
assert (
|
||||
normalized_usage({"prompt_tokens": 100, "prompt_tokens_details": {"cached_tokens": 30}})[
|
||||
"noncached_input_tokens"
|
||||
]
|
||||
== 70
|
||||
)
|
||||
raw = {"last_message": {"finished_reason": "exceed_max_iters"}, "result": "native answer"}
|
||||
native = AsyncMock(return_value=raw)
|
||||
wrapper = SimpleNamespace(reply=native)
|
||||
observe_completion(SimpleNamespace(context=SimpleNamespace(components={"agent_wrapper": {"bench": wrapper}})))
|
||||
assert await wrapper.reply("unchanged input", max_iters=30) is raw
|
||||
native.assert_awaited_once_with("unchanged input", max_iters=30)
|
||||
native.return_value = {"last_message": {"finished_reason": "interrupted"}}
|
||||
with pytest.raises(RuntimeError, match="interrupted"):
|
||||
await wrapper.reply("unchanged input")
|
||||
85
tests/unit/test_mobilemem_judge.py
Normal file
85
tests/unit/test_mobilemem_judge.py
Normal file
|
|
@ -0,0 +1,85 @@
|
|||
"""Standalone judge Jobs preserve official prompts, parsers and generation defaults."""
|
||||
|
||||
# pylint: disable=missing-function-docstring,protected-access
|
||||
import hashlib
|
||||
from importlib import import_module
|
||||
from pathlib import Path
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
from agentscope.message import TextBlock
|
||||
from agentscope.model import ChatResponse, ChatUsage
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
from reme.components.runtime_context import RuntimeContext
|
||||
from reme.plugin_manifest import parse_plugin_manifest
|
||||
|
||||
|
||||
@pytest.fixture(name="judge")
|
||||
def fixture_judge(monkeypatch):
|
||||
name = "mobilemem"
|
||||
root = Path(__file__).resolve().parents[2] / "plugins" / f"{name}-judge" / "src"
|
||||
monkeypatch.syspath_prepend(str(root))
|
||||
module = import_module(f"judge_{name}.llm_judge")
|
||||
prefix = "MobileMem"
|
||||
step = getattr(module, f"{prefix}JudgeStep")()
|
||||
step.context = RuntimeContext.from_context(
|
||||
None,
|
||||
question="QUESTION",
|
||||
reference="REFERENCE",
|
||||
prediction="PREDICTION",
|
||||
evidence=["first", "second"],
|
||||
category="information_extraction",
|
||||
)
|
||||
step.as_llm = AsyncMock(
|
||||
return_value=ChatResponse(
|
||||
content=[TextBlock(text='{"label":"CORRECT"}')],
|
||||
is_last=True,
|
||||
usage=ChatUsage(100, 10, 0.01, cache_input_tokens=30),
|
||||
),
|
||||
)
|
||||
return name, step, import_module(f"judge_{name}.protocol"), root
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_judge_job_uses_only_official_prompt_and_separate_usage(judge):
|
||||
name, step, protocol, root = judge
|
||||
response = await step.execute()
|
||||
assert response.success and response.metadata["llm_judge_score"] == 1
|
||||
assert response.metadata["usage"]["cache_input_tokens"] == 30
|
||||
kwargs = {"evidence": ["first", "second"]}
|
||||
expected = protocol.judge_prompt("QUESTION", "REFERENCE", "PREDICTION", **kwargs)
|
||||
assert step.as_llm.call_args.args[0][0].get_text_content() == expected
|
||||
assert len(step.as_llm.call_args.args[0]) == 1 and not step.as_llm.call_args.kwargs
|
||||
text = (root / f"judge_{name}" / "plugin.yaml").read_text()
|
||||
parse_plugin_manifest(text, plugin_name=f"{name}-judge")
|
||||
manifest = yaml.safe_load(text)["application_defaults"]
|
||||
assert manifest["jobs"]["answer_judge"]["steps"] == [{"backend": f"{name}_judge_step", "as_llm": "judge"}]
|
||||
model = manifest["components"]["as_llm"]["judge"]
|
||||
assert model["extra_body"] == {"enable_thinking": False} and model["parameters"]["temperature"] == 0
|
||||
assert model["parameters"]["max_tokens"] == 4096
|
||||
step.as_llm.return_value = ChatResponse(content=[TextBlock(text="malformed")], is_last=True)
|
||||
assert (await step.execute()).metadata["parse_default_zero"] is True
|
||||
step.as_llm.side_effect = TimeoutError("provider")
|
||||
with pytest.raises(TimeoutError):
|
||||
await step.execute()
|
||||
|
||||
|
||||
def test_mobilemem_official_prompt_and_parser_golden_contracts(monkeypatch):
|
||||
monkeypatch.syspath_prepend(str(Path(__file__).resolve().parents[2] / "plugins/mobilemem-judge/src"))
|
||||
protocol = import_module("judge_mobilemem.protocol")
|
||||
for mode, digest in [
|
||||
("membase_default", "7880f17ac2335307094dc8b02fa2c0093e1bec7b335215e3362e95a785a0f495"),
|
||||
("native_strict", "8710fcee94ffaa1b40911073b1f6404b62b93d36a4cc8236f3a10fef73622ca7"),
|
||||
]:
|
||||
prompt = protocol.judge_prompt(
|
||||
"QUESTION $prediction",
|
||||
"REFERENCE",
|
||||
"PREDICTION $question",
|
||||
["first", "second"],
|
||||
judge_protocol=mode,
|
||||
)
|
||||
assert hashlib.sha256(prompt.encode()).hexdigest() == digest
|
||||
for raw, score, default in [('{"label":"CORRECT"}', 1, False), ("CORRECT WRONG", 0, False), ("bad", 0, True)]:
|
||||
parsed = protocol.parse_judge(raw)
|
||||
assert (parsed["llm_judge_score"], parsed["parse_default_zero"]) == (score, default)
|
||||
Loading…
Add table
Reference in a new issue