mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-09-07 08:26:06 +00:00
halumem和longmemeval的Benchmark评估代码 (#124)
* feat(benchmark): 添加 LongMemEval 评估功能和内存检索器 - 实现了 LongMemEval 数据集的评估管道 - 添加了 PersonalLongmemevalRetriever 和 PersonalLongmemevalSummarizer - 创建了详细的统计分析工具 compute_stats.py - 实现了完整的答案判断和准确性计算功能 - 集成了 ReMe 内存操作和查询功能 - 添加了性能指标和时间统计功能 * feat(benchmark): 添加内存准确性和完整性评估功能 - 实现了 evaluation_for_memory_accuracy 函数用于评估提取内存的准确性 - 实现了 evaluation_for_memory_integrity 函数用于评估内存完整性 - 创建了 MemoryIntegrityEvaluator 类来评估内存点覆盖情况 - 创建了 MemoryAccuracyEvaluator 类来评估提取内存的准确性 - 添加了 compute_memory_integrity_metrics 和 compute_memory_accuracy_metrics 统计函数 - 在 MetricsAggregator 中集成内存完整性和准确性指标计算 - 更新了命令行参数默认值：top_k 改为 10，batch_size 改为 16 - 重构了个人记忆汇总器中的工具循环逻辑 - 更新了评估提示词模板以支持内存质量评估 - 添加了新的检索配置和模型设置 * feat(memory): 添加个人记忆摘要器配置文件 - 新增 personal_halumem_summarizer_adddraft.yaml 配置文件 - 新增 personal_halumem_summarizer_original_backup.yaml 备份配置文件 - 实现记忆架构师系统提示和用户消息模板 - 实现个人资料代理系统提示和用户消息模板 - 支持生物特征和行为模式记忆存储 - 实现记忆去重和合并功能 - 支持用户个人资料的动态更新和删除操作 * feat(memory): 添加个人记忆摘要器配置文件 - 新增 personal_halumem_summarizer_adddraft.yaml 配置文件 - 新增 personal_halumem_summarizer_original_backup.yaml 备份配置文件 - 实现记忆架构师系统提示和用户消息模板 - 实现个人资料代理系统提示和用户消息模板 - 支持生物特征和行为模式记忆存储 - 实现记忆去重和合并功能 - 支持用户个人资料的动态更新和删除操作 * docs(readme): 添加 ReMe Memory Agent 详细介绍文档 - 创建英文版 README.md 包含核心概念、架构设计和使用指南 - 创建中文版 README_ZH.md 提供完整的本地化文档 - 介绍 Agent 驱动的记忆管理理念和层次化检索机制 - 详述项目架构包括 ReMeSummarizer 和 ReMeRetriever 组件 - 提供快速开始示例和程序化内存操作方法 - 展示 LoCoMo、LongMemEval、HaluMem 基准测试结果 - 包含完整的项目结构说明和配置要求 * chore(config): 移除配置文件中的API密钥 - 从配置文件中删除FLOW_LLM_API_KEY环境变量设置 - 移除相关的API密钥配置项 - 更新配置文档以反映新的安全实践 - 确保敏感信息不再硬编码在配置文件中 - 添加注释说明如何通过环境变量方式配置API密钥 * fix(benchmark): 修复模型调用和配置参数问题 - 修正了reme.get_llm方法的参数传递，移除冗余的name参数 - 添加了qwen3-max模型的配置支持 - 调整了默认并发数从16降至4以提高稳定性 - 修改算法版本默认值从longmemeval和v1统一为default - 减少每类样本数量默认值从16至2以优化测试效率 * feat(benchmark): 添加记忆准确性和完整性评估功能 - 修改了 simple_request_for_json 调用以支持模型名称参数 - 新增 evaluation_for_memory_accuracy 函数用于评估记忆准确性 - 新增 evaluation_for_memory_integrity 函数用于评估记忆完整性 - 将默认模型名称从 qwen3-max 更改为 None - 更新提取记忆逻辑以过滤 time_int 和 when_to_use 字段 - 新增 MemoryIntegrityEvaluator 类用于评估记忆完整性 - 新增 MemoryAccuracyEvaluator 类用于评估记忆准确性 - 添加 compute_memory_integrity_metrics 方法计算记忆完整性指标 - 添加 compute_memory_accuracy_metrics 方法计算记忆准确性指标 - 配置多种新 LLM 模型包括 qwen-plus-t、qwen-max-t、gpt-4o-mini 等 - 初始化完整性评估器和准确性评估器实例 - 在会话数据中添加记忆完整性和准确性评估结果 - 收集记忆完整性记录和准确性记录用于统计 - 在最终结果中包含记忆完整性和准确性指标 - 更新摘要打印方法显示记忆完整性和准确性统计信息 - 更新默认评估模型为 gpt-4o-mini-2024-07-18 * docs(readme): 删除 ReMe Memory Agent 的中英文文档 - 移除英文版 README.md 中关于 ReMe Memory Agent 的详细介绍 - 删除中文版 README_ZH.md 中关于 ReMe Memory Agent 的完整文档 - 清理了包括架构图、功能特性、快速开始和实验数据在内的所有文档内容
This commit is contained in:
Zhouwk
GitHub
parent
3dcb9e09f4
commit
eff323105f
8 changed files with 3696 additions and 43 deletions
|
|
@ -223,9 +223,90 @@ async def answer_question_with_memories(
|
|||
question=question,
|
||||
)
|
||||
|
||||
result = await reme.get_llm("qwen3_max_instruct").simple_request_for_json(
|
||||
result = await reme.get_llm(model_name).simple_request_for_json(
|
||||
prompt=prompt,
|
||||
model_name=model_name,
|
||||
model_name=None,
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
async def evaluation_for_memory_accuracy(
|
||||
reme: ReMe,
|
||||
dialogue: str,
|
||||
golden_memories: list[dict],
|
||||
candidate_memory: dict,
|
||||
):
|
||||
"""
|
||||
Memory Accuracy Evaluation - Check if an extracted memory is accurate.
|
||||
|
||||
Args:
|
||||
reme: ReMe instance with default_llm and prompt_handler
|
||||
dialogue: The formatted dialogue string
|
||||
golden_memories: List of golden memory points from the session
|
||||
candidate_memory: The extracted memory to evaluate
|
||||
model_name: Model name to use for LLM request
|
||||
|
||||
Returns:
|
||||
dict with 'accuracy_score' (0/1/2), 'is_included_in_golden_memories' (true/false), and 'reason'
|
||||
"""
|
||||
# Format golden memories as string
|
||||
golden_memories_text = "\n".join(
|
||||
[f"- {m.get('memory_content', str(m))}" for m in golden_memories],
|
||||
)
|
||||
|
||||
# Extract candidate memory content
|
||||
candidate_content = candidate_memory.get("content", candidate_memory.get("memory_content", str(candidate_memory)))
|
||||
|
||||
prompt = reme.prompt_handler.prompt_format(
|
||||
"EVALUATION_PROMPT_FOR_MEMORY_ACCURACY",
|
||||
dialogue=dialogue,
|
||||
golden_memories=golden_memories_text,
|
||||
candidate_memory=candidate_content,
|
||||
)
|
||||
|
||||
result = await reme.get_llm("qwen-flash").simple_request_for_json(
|
||||
prompt=prompt,
|
||||
model_name=None,
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
async def evaluation_for_memory_integrity(
|
||||
reme: ReMe,
|
||||
extracted_memories: list[dict],
|
||||
expected_memory_point: dict,
|
||||
):
|
||||
"""
|
||||
Memory Integrity Evaluation - Check if extracted memories cover the expected memory point.
|
||||
|
||||
Args:
|
||||
reme: ReMe instance with default_llm and prompt_handler
|
||||
extracted_memories: List of extracted memory dicts
|
||||
expected_memory_point: The expected memory point dict with 'memory_content' field
|
||||
model_name: Model name to use for LLM request
|
||||
|
||||
Returns:
|
||||
dict with 'reasoning' and 'score' fields (score: 0, 1, or 2)
|
||||
"""
|
||||
# Format extracted memories as string
|
||||
memories_text = "\n".join(
|
||||
[f"- {m.get('content', m.get('memory_content', str(m)))}" for m in extracted_memories],
|
||||
)
|
||||
|
||||
# Extract expected memory point content
|
||||
expected_content = expected_memory_point.get("memory_content", str(expected_memory_point))
|
||||
|
||||
prompt = reme.prompt_handler.prompt_format(
|
||||
"EVALUATION_PROMPT_FOR_MEMORY_INTEGRITY",
|
||||
memories=memories_text,
|
||||
expected_memory_point=expected_content,
|
||||
)
|
||||
|
||||
result = await reme.get_llm("qwen-flash").simple_request_for_json(
|
||||
prompt=prompt,
|
||||
model_name=None,
|
||||
)
|
||||
|
||||
return result
|
||||
|
|
@ -238,7 +319,7 @@ async def evaluation_for_question(
|
|||
key_memory_points: str,
|
||||
response: str,
|
||||
dialogue: str = None,
|
||||
model_name: str = "qwen3-max",
|
||||
model_name: str = None,
|
||||
):
|
||||
"""
|
||||
Question-Answering Evaluation with optional Dialogue Context.
|
||||
|
|
@ -325,7 +406,13 @@ class MemoryProcessor:
|
|||
duration_ms = (time.time() - start) * 1000
|
||||
total_duration_ms += duration_ms
|
||||
|
||||
extracted_memories.extend([m.model_dump(exclude_none=True) for m in result["answer"]])
|
||||
extracted_memories.extend(
|
||||
[
|
||||
memory_node.model_dump(exclude_none=True)
|
||||
for memory_node in result["answer"]
|
||||
if "time_int" in memory_node.metadata and memory_node.when_to_use == ""
|
||||
],
|
||||
)
|
||||
summary_messages.extend([m.simple_dump(enable_argument_dict=True) for m in result["messages"]])
|
||||
|
||||
return extracted_memories, summary_messages, total_duration_ms
|
||||
|
|
@ -457,6 +544,95 @@ class QuestionAnsweringEvaluator:
|
|||
return results
|
||||
|
||||
|
||||
class MemoryIntegrityEvaluator:
|
||||
"""Evaluates memory integrity - whether extracted memories cover expected memory points."""
|
||||
|
||||
def __init__(self, reme: ReMe, eval_model_name: str = "gpt-4o-mini"):
|
||||
self.reme = reme
|
||||
self.eval_model_name = eval_model_name
|
||||
|
||||
async def evaluate_memory_points(
|
||||
self,
|
||||
extracted_memories: list[dict],
|
||||
memory_points: list[dict],
|
||||
) -> list[dict]:
|
||||
"""
|
||||
Evaluate whether extracted memories cover each expected memory point.
|
||||
|
||||
Args:
|
||||
extracted_memories: List of memories extracted by the system
|
||||
memory_points: List of expected memory points from the session
|
||||
|
||||
Returns:
|
||||
List of evaluation results, one per memory point
|
||||
"""
|
||||
results = []
|
||||
|
||||
for memory_point in memory_points:
|
||||
eval_result = await evaluation_for_memory_integrity(
|
||||
reme=self.reme,
|
||||
extracted_memories=extracted_memories,
|
||||
expected_memory_point=memory_point,
|
||||
)
|
||||
|
||||
# Build result record
|
||||
integrity_result = {
|
||||
**memory_point,
|
||||
"integrity_score": eval_result.get("score"),
|
||||
"integrity_reasoning": eval_result.get("reasoning", ""),
|
||||
}
|
||||
results.append(integrity_result)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
class MemoryAccuracyEvaluator:
|
||||
"""Evaluates memory accuracy - whether each extracted memory is accurate."""
|
||||
|
||||
def __init__(self, reme: ReMe, eval_model_name: str = "gpt-4o-mini"):
|
||||
self.reme = reme
|
||||
self.eval_model_name = eval_model_name
|
||||
|
||||
async def evaluate_extracted_memories(
|
||||
self,
|
||||
extracted_memories: list[dict],
|
||||
memory_points: list[dict],
|
||||
formatted_dialogue: str,
|
||||
) -> list[dict]:
|
||||
"""
|
||||
Evaluate the accuracy of each extracted memory.
|
||||
|
||||
Args:
|
||||
extracted_memories: List of memories extracted by the system
|
||||
memory_points: List of golden memory points from the session
|
||||
formatted_dialogue: The formatted dialogue string
|
||||
|
||||
Returns:
|
||||
List of evaluation results, one per extracted memory
|
||||
"""
|
||||
results = []
|
||||
|
||||
for memory in extracted_memories:
|
||||
eval_result = await evaluation_for_memory_accuracy(
|
||||
reme=self.reme,
|
||||
dialogue=formatted_dialogue,
|
||||
golden_memories=memory_points,
|
||||
candidate_memory=memory,
|
||||
)
|
||||
|
||||
# Build result record
|
||||
accuracy_result = {
|
||||
"memory_content": memory.get("content", memory.get("memory_content", str(memory))),
|
||||
"memory_id": memory.get("memory_id", ""),
|
||||
"accuracy_score": eval_result.get("accuracy_score"),
|
||||
"is_included_in_golden_memories": eval_result.get("is_included_in_golden_memories"),
|
||||
"accuracy_reason": eval_result.get("reason", ""),
|
||||
}
|
||||
results.append(accuracy_result)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
class MetricsAggregator:
|
||||
"""Aggregates evaluation metrics."""
|
||||
|
||||
|
|
@ -528,6 +704,148 @@ class MetricsAggregator:
|
|||
"with_original_memories": MetricsAggregator._compute_single_metric(qa_records, "original_result_type"),
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def compute_memory_integrity_metrics(integrity_records: list[dict]) -> dict[str, Any]:
|
||||
"""
|
||||
Compute memory integrity metrics.
|
||||
|
||||
Args:
|
||||
integrity_records: List of integrity evaluation results
|
||||
|
||||
Returns:
|
||||
dict with integrity metrics (score distribution and average)
|
||||
"""
|
||||
total = len(integrity_records)
|
||||
if total == 0:
|
||||
return {
|
||||
"total_memory_points": 0,
|
||||
"score_2_count": 0,
|
||||
"score_1_count": 0,
|
||||
"score_0_count": 0,
|
||||
"score_2_ratio": 0,
|
||||
"score_1_ratio": 0,
|
||||
"score_0_ratio": 0,
|
||||
"average_score": 0,
|
||||
"valid_count": 0,
|
||||
}
|
||||
|
||||
score_2_count = 0
|
||||
score_1_count = 0
|
||||
score_0_count = 0
|
||||
valid_count = 0
|
||||
total_score = 0
|
||||
|
||||
for record in integrity_records:
|
||||
score = record.get("integrity_score")
|
||||
# Handle both string and int scores
|
||||
if score is not None:
|
||||
try:
|
||||
score_int = int(score)
|
||||
valid_count += 1
|
||||
total_score += score_int
|
||||
if score_int == 2:
|
||||
score_2_count += 1
|
||||
elif score_int == 1:
|
||||
score_1_count += 1
|
||||
elif score_int == 0:
|
||||
score_0_count += 1
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
metrics = {
|
||||
"total_memory_points": total,
|
||||
"score_2_count": score_2_count,
|
||||
"score_1_count": score_1_count,
|
||||
"score_0_count": score_0_count,
|
||||
"score_2_ratio": score_2_count / total if total > 0 else 0,
|
||||
"score_1_ratio": score_1_count / total if total > 0 else 0,
|
||||
"score_0_ratio": score_0_count / total if total > 0 else 0,
|
||||
"average_score": total_score / valid_count if valid_count > 0 else 0,
|
||||
"accuracy": score_2_count / valid_count if valid_count > 0 else 0,
|
||||
"valid_count": valid_count,
|
||||
}
|
||||
|
||||
return metrics
|
||||
|
||||
@staticmethod
|
||||
def compute_memory_accuracy_metrics(accuracy_records: list[dict]) -> dict[str, Any]:
|
||||
"""
|
||||
Compute memory accuracy metrics for extracted memories.
|
||||
|
||||
Args:
|
||||
accuracy_records: List of accuracy evaluation results
|
||||
|
||||
Returns:
|
||||
dict with accuracy metrics (score distribution, average, and inclusion ratio)
|
||||
"""
|
||||
total = len(accuracy_records)
|
||||
if total == 0:
|
||||
return {
|
||||
"total_extracted_memories": 0,
|
||||
"score_2_count": 0,
|
||||
"score_1_count": 0,
|
||||
"score_0_count": 0,
|
||||
"score_2_ratio": 0,
|
||||
"score_1_ratio": 0,
|
||||
"score_0_ratio": 0,
|
||||
"average_score": 0,
|
||||
"accuracy": 0,
|
||||
"included_in_golden_count": 0,
|
||||
"included_in_golden_ratio": 0,
|
||||
"valid_count": 0,
|
||||
}
|
||||
|
||||
score_2_count = 0
|
||||
score_1_count = 0
|
||||
score_0_count = 0
|
||||
included_count = 0
|
||||
valid_count = 0
|
||||
total_score = 0
|
||||
|
||||
for record in accuracy_records:
|
||||
score = record.get("accuracy_score")
|
||||
included = record.get("is_included_in_golden_memories")
|
||||
|
||||
# Handle both string and int scores
|
||||
if score is not None:
|
||||
try:
|
||||
score_int = int(score)
|
||||
valid_count += 1
|
||||
total_score += score_int
|
||||
if score_int == 2:
|
||||
score_2_count += 1
|
||||
elif score_int == 1:
|
||||
score_1_count += 1
|
||||
elif score_int == 0:
|
||||
score_0_count += 1
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
# Handle is_included_in_golden_memories
|
||||
if included is not None:
|
||||
if isinstance(included, bool):
|
||||
if included:
|
||||
included_count += 1
|
||||
elif isinstance(included, str) and included.lower() == "true":
|
||||
included_count += 1
|
||||
|
||||
metrics = {
|
||||
"total_extracted_memories": total,
|
||||
"score_2_count": score_2_count,
|
||||
"score_1_count": score_1_count,
|
||||
"score_0_count": score_0_count,
|
||||
"score_2_ratio": score_2_count / total if total > 0 else 0,
|
||||
"score_1_ratio": score_1_count / total if total > 0 else 0,
|
||||
"score_0_ratio": score_0_count / total if total > 0 else 0,
|
||||
"average_score": total_score / valid_count if valid_count > 0 else 0,
|
||||
"accuracy": score_2_count / valid_count if valid_count > 0 else 0,
|
||||
"included_in_golden_count": included_count,
|
||||
"included_in_golden_ratio": included_count / total if total > 0 else 0,
|
||||
"valid_count": valid_count,
|
||||
}
|
||||
|
||||
return metrics
|
||||
|
||||
@staticmethod
|
||||
def compute_time_metrics(eval_results_file: str) -> dict[str, float]:
|
||||
"""Compute timing metrics from evaluation results."""
|
||||
|
|
@ -567,6 +885,34 @@ class HaluMemEvaluator:
|
|||
default_llm_config={
|
||||
"model_name": self.config.reme_model_name,
|
||||
},
|
||||
llms={
|
||||
"qwen-plus-t": {
|
||||
"backend": "openai",
|
||||
"model_name": "qwen-plus",
|
||||
"extra_body": {
|
||||
"enable_thinking": True,
|
||||
},
|
||||
},
|
||||
"qwen-max-t": {
|
||||
"backend": "openai",
|
||||
"model_name": "qwen3-max",
|
||||
"extra_body": {
|
||||
"enable_thinking": True,
|
||||
},
|
||||
},
|
||||
"gpt-4o-mini": {
|
||||
"backend": "openai",
|
||||
"model_name": "gpt-4o-mini-2024-07-18",
|
||||
},
|
||||
"gpt-4o-mini-2024-07-18": {
|
||||
"backend": "openai",
|
||||
"model_name": "gpt-4o-mini-2024-07-18",
|
||||
},
|
||||
"qwen-flash": {
|
||||
"backend": "openai",
|
||||
"model_name": "qwen-flash",
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
# Load evaluation prompts into ReMe's prompt handler
|
||||
|
|
@ -587,6 +933,14 @@ class HaluMemEvaluator:
|
|||
config.top_k,
|
||||
config.eval_model_name,
|
||||
)
|
||||
self.integrity_evaluator = MemoryIntegrityEvaluator(
|
||||
self.reme,
|
||||
eval_model_name="qwen-flash",
|
||||
)
|
||||
self.accuracy_evaluator = MemoryAccuracyEvaluator(
|
||||
self.reme,
|
||||
eval_model_name="qwen-flash",
|
||||
)
|
||||
self.data_loader = DataLoader()
|
||||
|
||||
# For real-time updates
|
||||
|
|
@ -641,6 +995,26 @@ class HaluMemEvaluator:
|
|||
},
|
||||
)
|
||||
|
||||
# Evaluate memory integrity - check if extracted memories cover memory points
|
||||
memory_points = session.get("memory_points", [])
|
||||
formatted_dialogue = self.data_loader.format_dialogue_for_eval(dialogue, user_name)
|
||||
|
||||
if memory_points and extracted_memories:
|
||||
integrity_results = await self.integrity_evaluator.evaluate_memory_points(
|
||||
extracted_memories=extracted_memories,
|
||||
memory_points=memory_points,
|
||||
)
|
||||
session_data["memory_integrity_results"] = integrity_results
|
||||
|
||||
# Evaluate memory accuracy - check if each extracted memory is accurate
|
||||
if extracted_memories and memory_points:
|
||||
accuracy_results = await self.accuracy_evaluator.evaluate_extracted_memories(
|
||||
extracted_memories=extracted_memories,
|
||||
memory_points=memory_points,
|
||||
formatted_dialogue=formatted_dialogue,
|
||||
)
|
||||
session_data["memory_accuracy_results"] = accuracy_results
|
||||
|
||||
# Evaluate questions if present
|
||||
if "questions" in session:
|
||||
formatted_dialogue = self.data_loader.format_dialogue_for_eval(dialogue, user_name)
|
||||
|
|
@ -762,8 +1136,10 @@ class HaluMemEvaluator:
|
|||
if not os.path.exists(results_file):
|
||||
return
|
||||
|
||||
# Collect all QA records
|
||||
# Collect all QA records, memory integrity records, and accuracy records
|
||||
qa_records = []
|
||||
integrity_records = []
|
||||
accuracy_records = []
|
||||
try:
|
||||
with open(results_file, "r", encoding="utf-8") as f:
|
||||
for line in f:
|
||||
|
|
@ -779,6 +1155,16 @@ class HaluMemEvaluator:
|
|||
qa_records.extend(
|
||||
eval_results.get("question_answering_records", []),
|
||||
)
|
||||
|
||||
# Collect memory integrity records
|
||||
integrity_records.extend(
|
||||
session.get("memory_integrity_results", []),
|
||||
)
|
||||
|
||||
# Collect memory accuracy records
|
||||
accuracy_records.extend(
|
||||
session.get("memory_accuracy_results", []),
|
||||
)
|
||||
except (json.JSONDecodeError, KeyError):
|
||||
return
|
||||
|
||||
|
|
@ -788,13 +1174,19 @@ class HaluMemEvaluator:
|
|||
# Compute metrics
|
||||
qa_metrics = MetricsAggregator.compute_qa_metrics(qa_records)
|
||||
time_metrics = MetricsAggregator.compute_time_metrics(results_file)
|
||||
integrity_metrics = MetricsAggregator.compute_memory_integrity_metrics(integrity_records)
|
||||
accuracy_metrics = MetricsAggregator.compute_memory_accuracy_metrics(accuracy_records)
|
||||
|
||||
final_results = {
|
||||
"overall_score": {
|
||||
"question_answering": qa_metrics,
|
||||
"memory_integrity": integrity_metrics,
|
||||
"memory_accuracy": accuracy_metrics,
|
||||
"time_consuming": time_metrics,
|
||||
},
|
||||
"question_answering_records": qa_records,
|
||||
"memory_integrity_records": integrity_records,
|
||||
"memory_accuracy_records": accuracy_records,
|
||||
}
|
||||
|
||||
# Save statistics
|
||||
|
|
@ -810,6 +1202,8 @@ class HaluMemEvaluator:
|
|||
|
||||
# Collect all QA records
|
||||
qa_records = []
|
||||
integrity_records = []
|
||||
accuracy_records = []
|
||||
with open(results_file, "r", encoding="utf-8") as f:
|
||||
for line in f:
|
||||
if not line.strip():
|
||||
|
|
@ -825,16 +1219,32 @@ class HaluMemEvaluator:
|
|||
eval_results.get("question_answering_records", []),
|
||||
)
|
||||
|
||||
# Collect memory integrity records
|
||||
integrity_records.extend(
|
||||
session.get("memory_integrity_results", []),
|
||||
)
|
||||
|
||||
# Collect memory accuracy records
|
||||
accuracy_records.extend(
|
||||
session.get("memory_accuracy_results", []),
|
||||
)
|
||||
|
||||
# Compute metrics
|
||||
qa_metrics = MetricsAggregator.compute_qa_metrics(qa_records)
|
||||
time_metrics = MetricsAggregator.compute_time_metrics(results_file)
|
||||
integrity_metrics = MetricsAggregator.compute_memory_integrity_metrics(integrity_records)
|
||||
accuracy_metrics = MetricsAggregator.compute_memory_accuracy_metrics(accuracy_records)
|
||||
|
||||
final_results = {
|
||||
"overall_score": {
|
||||
"question_answering": qa_metrics,
|
||||
"memory_integrity": integrity_metrics,
|
||||
"memory_accuracy": accuracy_metrics,
|
||||
"time_consuming": time_metrics,
|
||||
},
|
||||
"question_answering_records": qa_records,
|
||||
"memory_integrity_records": integrity_records,
|
||||
"memory_accuracy_records": accuracy_records,
|
||||
}
|
||||
|
||||
# Save final report
|
||||
|
|
@ -845,35 +1255,95 @@ class HaluMemEvaluator:
|
|||
print(f"📊 Statistics saved to: {report_file}\n")
|
||||
|
||||
# Print summary
|
||||
self._print_summary(qa_metrics, time_metrics)
|
||||
self._print_summary(qa_metrics, time_metrics, integrity_metrics, accuracy_metrics)
|
||||
|
||||
def _print_summary(self, qa_metrics: dict, time_metrics: dict):
|
||||
def _print_summary(
|
||||
self,
|
||||
qa_metrics: dict,
|
||||
time_metrics: dict,
|
||||
integrity_metrics: dict = None,
|
||||
accuracy_metrics: dict = None,
|
||||
):
|
||||
"""Print evaluation summary."""
|
||||
print("=" * 80)
|
||||
print("EVALUATION SUMMARY - REME")
|
||||
print("=" * 80 + "\n")
|
||||
|
||||
# Print metrics for LLM-generated answer (result_type)
|
||||
llm_metrics = qa_metrics["with_llm_answer"]
|
||||
print("📊 Question Answering (with LLM answer):")
|
||||
print(f" Correct (all): {llm_metrics['correct_qa_ratio(all)']:.4f}")
|
||||
print(f" Hallucination (all): {llm_metrics['hallucination_qa_ratio(all)']:.4f}")
|
||||
print(f" Omission (all): {llm_metrics['omission_qa_ratio(all)']:.4f}")
|
||||
print(f" Correct (valid): {llm_metrics['correct_qa_ratio(valid)']:.4f}")
|
||||
print(f" Hallucination (valid): {llm_metrics['hallucination_qa_ratio(valid)']:.4f}")
|
||||
print(f" Omission (valid): {llm_metrics['omission_qa_ratio(valid)']:.4f}")
|
||||
print(f" Valid/Total: {llm_metrics['qa_valid_num']}/{llm_metrics['qa_num']}")
|
||||
# Print memory integrity metrics
|
||||
if integrity_metrics and integrity_metrics.get("total_memory_points", 0) > 0:
|
||||
print("🧠 Memory Integrity (coverage of expected memory points):")
|
||||
total = integrity_metrics["total_memory_points"]
|
||||
print(
|
||||
f" Score 2 (Fully covered): "
|
||||
f"{integrity_metrics['score_2_count']}/{total} "
|
||||
f"({integrity_metrics['score_2_ratio']:.4f})",
|
||||
)
|
||||
print(
|
||||
f" Score 1 (Partially covered): "
|
||||
f"{integrity_metrics['score_1_count']}/{total} "
|
||||
f"({integrity_metrics['score_1_ratio']:.4f})",
|
||||
)
|
||||
print(
|
||||
f" Score 0 (Not covered): "
|
||||
f"{integrity_metrics['score_0_count']}/{total} "
|
||||
f"({integrity_metrics['score_0_ratio']:.4f})",
|
||||
)
|
||||
print(f" Average Score: {integrity_metrics['average_score']:.4f}")
|
||||
print(f" Accuracy (score=2 ratio): {integrity_metrics['accuracy']:.4f}")
|
||||
print(f" Valid/Total: " f"{integrity_metrics['valid_count']}/{total}")
|
||||
print()
|
||||
|
||||
# Print metrics for original retrieved memories (original_result_type)
|
||||
orig_metrics = qa_metrics["with_original_memories"]
|
||||
print("\n📊 Question Answering (with original memories):")
|
||||
print(f" Correct (all): {orig_metrics['correct_qa_ratio(all)']:.4f}")
|
||||
print(f" Hallucination (all): {orig_metrics['hallucination_qa_ratio(all)']:.4f}")
|
||||
print(f" Omission (all): {orig_metrics['omission_qa_ratio(all)']:.4f}")
|
||||
print(f" Correct (valid): {orig_metrics['correct_qa_ratio(valid)']:.4f}")
|
||||
print(f" Hallucination (valid): {orig_metrics['hallucination_qa_ratio(valid)']:.4f}")
|
||||
print(f" Omission (valid): {orig_metrics['omission_qa_ratio(valid)']:.4f}")
|
||||
print(f" Valid/Total: {orig_metrics['qa_valid_num']}/{orig_metrics['qa_num']}")
|
||||
# Print memory accuracy metrics
|
||||
if accuracy_metrics and accuracy_metrics.get("total_extracted_memories", 0) > 0:
|
||||
print("🎯 Memory Accuracy (accuracy of extracted memories):")
|
||||
total_acc = accuracy_metrics["total_extracted_memories"]
|
||||
print(
|
||||
f" Score 2 (Fully accurate): "
|
||||
f"{accuracy_metrics['score_2_count']}/{total_acc} "
|
||||
f"({accuracy_metrics['score_2_ratio']:.4f})",
|
||||
)
|
||||
print(
|
||||
f" Score 1 (Partially accurate): "
|
||||
f"{accuracy_metrics['score_1_count']}/{total_acc} "
|
||||
f"({accuracy_metrics['score_1_ratio']:.4f})",
|
||||
)
|
||||
print(
|
||||
f" Score 0 (Hallucinated): "
|
||||
f"{accuracy_metrics['score_0_count']}/{total_acc} "
|
||||
f"({accuracy_metrics['score_0_ratio']:.4f})",
|
||||
)
|
||||
print(f" Average Score: {accuracy_metrics['average_score']:.4f}")
|
||||
print(f" Accuracy (score=2 ratio): {accuracy_metrics['accuracy']:.4f}")
|
||||
print(
|
||||
f" Included in Golden: "
|
||||
f"{accuracy_metrics['included_in_golden_count']}/{total_acc} "
|
||||
f"({accuracy_metrics['included_in_golden_ratio']:.4f})",
|
||||
)
|
||||
print(f" Valid/Total: " f"{accuracy_metrics['valid_count']}/{total_acc}")
|
||||
print()
|
||||
|
||||
# Print metrics for LLM-generated answer (result_type)
|
||||
if qa_metrics and "with_llm_answer" in qa_metrics:
|
||||
llm_metrics = qa_metrics["with_llm_answer"]
|
||||
print("📊 Question Answering (with LLM answer):")
|
||||
print(f" Correct (all): {llm_metrics['correct_qa_ratio(all)']:.4f}")
|
||||
print(f" Hallucination (all): {llm_metrics['hallucination_qa_ratio(all)']:.4f}")
|
||||
print(f" Omission (all): {llm_metrics['omission_qa_ratio(all)']:.4f}")
|
||||
print(f" Correct (valid): {llm_metrics['correct_qa_ratio(valid)']:.4f}")
|
||||
print(f" Hallucination (valid): {llm_metrics['hallucination_qa_ratio(valid)']:.4f}")
|
||||
print(f" Omission (valid): {llm_metrics['omission_qa_ratio(valid)']:.4f}")
|
||||
print(f" Valid/Total: {llm_metrics['qa_valid_num']}/{llm_metrics['qa_num']}")
|
||||
|
||||
# Print metrics for original retrieved memories (original_result_type)
|
||||
orig_metrics = qa_metrics["with_original_memories"]
|
||||
print("\n📊 Question Answering (with original memories):")
|
||||
print(f" Correct (all): {orig_metrics['correct_qa_ratio(all)']:.4f}")
|
||||
print(f" Hallucination (all): {orig_metrics['hallucination_qa_ratio(all)']:.4f}")
|
||||
print(f" Omission (all): {orig_metrics['omission_qa_ratio(all)']:.4f}")
|
||||
print(f" Correct (valid): {orig_metrics['correct_qa_ratio(valid)']:.4f}")
|
||||
print(f" Hallucination (valid): {orig_metrics['hallucination_qa_ratio(valid)']:.4f}")
|
||||
print(f" Omission (valid): {orig_metrics['omission_qa_ratio(valid)']:.4f}")
|
||||
print(f" Valid/Total: {orig_metrics['qa_valid_num']}/{orig_metrics['qa_num']}")
|
||||
|
||||
print("\n⏱️ Time Metrics:")
|
||||
print(f" Memory Addition: {time_metrics['add_dialogue_duration_time']:.2f} min")
|
||||
|
|
@ -987,14 +1457,14 @@ if __name__ == "__main__":
|
|||
parser.add_argument(
|
||||
"--eval_model_name",
|
||||
type=str,
|
||||
default="qwen3-max",
|
||||
default="gpt-4o-mini-2024-07-18",
|
||||
help="Model name for evaluation (default: qwen3-max)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--algo_version",
|
||||
type=str,
|
||||
default="v1",
|
||||
help="Algorithm version for summary and retrieval (default: v1)",
|
||||
default="default",
|
||||
help="Algorithm version for summary and retrieval (default: default)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--enable_thinking_params",
|
||||
|
|
|
|||
|
|
@ -192,7 +192,7 @@ PROMPT_MEMOBASE: |
|
|||
|
||||
EVALUATION_PROMPT_FOR_MEMORY_INTEGRITY: |
|
||||
You are a strict **"Memory Integrity" evaluator**.
|
||||
Your core task is to assess whether an AI memory system has **missed any key memory points** after processing a conversation. This evaluation measures the system’s **memory integrity**, i.e., its ability to resist **amnesia** or **omission**.
|
||||
Your core task is to assess whether an AI memory system has **missed any key memory points** after processing a conversation. This evaluation measures the system's **memory integrity**, i.e., its ability to resist **amnesia** or **omission**.
|
||||
|
||||
# Evaluation Context & Data:
|
||||
|
||||
|
|
@ -212,13 +212,13 @@ EVALUATION_PROMPT_FOR_MEMORY_INTEGRITY: |
|
|||
# Scoring Rubric:
|
||||
|
||||
* **2:** Fully covered or implied.
|
||||
One or more items in “Extracted Memories” fully cover or logically imply all information in the “Expected Memory Point.”
|
||||
One or more items in "Extracted Memories" fully cover or logically imply all information in the "Expected Memory Point."
|
||||
|
||||
* **1:** Partially covered or mentioned.
|
||||
Some information in “Extracted Memories” mentions part of the “Expected Memory Point,” but key information is missing, inaccurate, or slightly incorrect.
|
||||
Some information in "Extracted Memories" mentions part of the "Expected Memory Point," but key information is missing, inaccurate, or slightly incorrect.
|
||||
|
||||
* **0:** Not mentioned or incorrect.
|
||||
“Extracted Memories” contains no mention of the “Expected Memory Point,” or the corresponding information is entirely wrong.
|
||||
"Extracted Memories" contains no mention of the "Expected Memory Point," or the corresponding information is entirely wrong.
|
||||
|
||||
# Scoring Notes:
|
||||
|
||||
|
|
@ -230,11 +230,11 @@ EVALUATION_PROMPT_FOR_MEMORY_INTEGRITY: |
|
|||
|
||||
* Semantic matching is acceptable; exact wording is **not** required.
|
||||
|
||||
* If “Extracted Memories” contains **conflicting information**, assign the **best possible coverage score** and mention the conflict in your reasoning.
|
||||
* If "Extracted Memories" contains **conflicting information**, assign the **best possible coverage score** and mention the conflict in your reasoning.
|
||||
|
||||
* Extra or stylistically different memories do **not** reduce the score; only the coverage of the **Expected Memory Point** matters.
|
||||
|
||||
* For uncertain wording (“might,” “probably,” “tends to,” etc.):
|
||||
* For uncertain wording ("might," "probably," "tends to," etc.):
|
||||
|
||||
* If the Expected Memory Point is a definite statement, usually assign **1 point**.
|
||||
|
||||
|
|
@ -283,10 +283,9 @@ EVALUATION_PROMPT_FOR_MEMORY_ACCURACY: |
|
|||
|
||||
* **2 points:** Every information point in the candidate memory is supported by the dialogue or golden memories, with **no contradictions or hallucinations**.
|
||||
* **1 point:** The candidate memory is *partially correct* (at least one supported information point) but also includes *unsupported* or *contradictory* content.
|
||||
* **0 points:** The candidate memory is **entirely unsupported or contradictory** to the sources (i.e., a “hallucinated memory”).
|
||||
* **0 points:** The candidate memory is **entirely unsupported or contradictory** to the sources (i.e., a "hallucinated memory").
|
||||
|
||||
> Note:
|
||||
>
|
||||
> * If a candidate memory contains multiple information points, **any unsupported or contradictory element** prevents a full score (2).
|
||||
> * If both supported and unsupported/conflicting content appear, assign a score of **1**.
|
||||
|
||||
|
|
@ -302,14 +301,14 @@ EVALUATION_PROMPT_FOR_MEMORY_ACCURACY: |
|
|||
* **true:**
|
||||
Every atomic information point in the candidate memory has a corresponding **field** in the golden memories (allowing for synonyms, paraphrases, or equivalent expressions; ignore value, polarity, or quantity differences).
|
||||
|
||||
* Note: A single field in the gold list may match multiple candidate points (e.g., multiple “drink preference” facts can be covered by one “drink preference” field in gold).
|
||||
* Note: A single field in the gold list may match multiple candidate points (e.g., multiple "drink preference" facts can be covered by one "drink preference" field in gold).
|
||||
* **false:**
|
||||
If **any** atomic information point’s field in the candidate memory cannot be found in the golden memories, mark as *false*.
|
||||
If **any** atomic information point's field in the candidate memory cannot be found in the golden memories, mark as *false*.
|
||||
|
||||
**Important Notes:**
|
||||
|
||||
* Field matching is restricted to fields that are **explicitly present or semantically recognizable** in the golden memories — no external knowledge may be used to expand the field set.
|
||||
* Differences in **values** (e.g., “Zhang San” vs. “Li Si”), **polarity** (like/dislike), or **exact number/time** do **not** affect this Boolean judgment.
|
||||
* Differences in **values** (e.g., "Zhang San" vs. "Li Si"), **polarity** (like/dislike), or **exact number/time** do **not** affect this Boolean judgment.
|
||||
|
||||
# Evaluation Procedure
|
||||
|
||||
|
|
@ -320,7 +319,7 @@ EVALUATION_PROMPT_FOR_MEMORY_ACCURACY: |
|
|||
3. Assign the **accuracy_score** (0 / 1 / 2) according to the rules above.
|
||||
4. Determine **is_included_in_golden_memories (true/false)**:
|
||||
|
||||
* Identify each information point’s field;
|
||||
* Identify each information point's field;
|
||||
* If *all* fields exist in the golden memories, mark as *true*; otherwise, *false*.
|
||||
5. Provide a **concise Chinese explanation** in `"reason"`, citing key evidence (short excerpts allowed), and clearly state any unsupported or contradictory parts if applicable.
|
||||
|
||||
|
|
@ -339,9 +338,9 @@ EVALUATION_PROMPT_FOR_MEMORY_ACCURACY: |
|
|||
|
||||
```json
|
||||
{{
|
||||
"reason": "Brief explanation in Chinese"
|
||||
"accuracy_score": "2 | 1 | 0",
|
||||
"is_included_in_golden_memories": "true | false",
|
||||
"reason": "Brief explanation in Chinese"
|
||||
}}
|
||||
```
|
||||
|
||||
|
|
|
|||
346
benchmark/longmemeval/compute_stats.py
Normal file
346
benchmark/longmemeval/compute_stats.py
Normal file
|
|
@ -0,0 +1,346 @@
|
|||
"""
|
||||
LongMemEval Evaluation Statistics Analyzer
|
||||
|
||||
Computes detailed statistics from evaluation results including:
|
||||
- Overall accuracy
|
||||
- Accuracy by question type
|
||||
- Timing statistics (summary, retrieval)
|
||||
- Memory extraction statistics
|
||||
|
||||
Usage:
|
||||
python bench/longmemeval/compute_stats.py \
|
||||
--results_dir bench/longmemeval/bench_results/longmemeval_reme
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
def load_results(results_dir: str) -> list[dict]:
|
||||
"""Load all question result files from the directory.
|
||||
|
||||
Args:
|
||||
results_dir: Path to the results directory
|
||||
|
||||
Returns:
|
||||
List of result dictionaries
|
||||
"""
|
||||
results_path = Path(results_dir)
|
||||
results = []
|
||||
|
||||
# Load individual question files
|
||||
question_files = sorted(results_path.glob("question_*.json"))
|
||||
|
||||
for file_path in question_files:
|
||||
try:
|
||||
with open(file_path, "r", encoding="utf-8") as f:
|
||||
result = json.load(f)
|
||||
results.append(result)
|
||||
except Exception as e:
|
||||
print(f"⚠️ Error loading {file_path}: {e}")
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def compute_accuracy_stats(results: list[dict]) -> dict[str, Any]:
|
||||
"""Compute overall and per-type accuracy statistics.
|
||||
|
||||
Args:
|
||||
results: List of result dictionaries
|
||||
|
||||
Returns:
|
||||
Dictionary with accuracy statistics
|
||||
"""
|
||||
total = len(results)
|
||||
correct = 0
|
||||
incorrect = 0
|
||||
error = 0
|
||||
|
||||
# Per question type statistics
|
||||
type_stats = defaultdict(lambda: {"total": 0, "correct": 0, "incorrect": 0, "error": 0})
|
||||
|
||||
for r in results:
|
||||
qtype = r.get("question_type", "unknown")
|
||||
judgment = r.get("judgment", {})
|
||||
is_correct = judgment.get("is_correct")
|
||||
|
||||
type_stats[qtype]["total"] += 1
|
||||
|
||||
if is_correct is True:
|
||||
correct += 1
|
||||
type_stats[qtype]["correct"] += 1
|
||||
elif is_correct is False:
|
||||
incorrect += 1
|
||||
type_stats[qtype]["incorrect"] += 1
|
||||
else:
|
||||
error += 1
|
||||
type_stats[qtype]["error"] += 1
|
||||
|
||||
# Compute accuracies
|
||||
overall = {
|
||||
"total": total,
|
||||
"correct": correct,
|
||||
"incorrect": incorrect,
|
||||
"error": error,
|
||||
"accuracy": correct / total if total > 0 else 0,
|
||||
"accuracy_valid": correct / (correct + incorrect) if (correct + incorrect) > 0 else 0,
|
||||
}
|
||||
|
||||
by_type = {}
|
||||
for qtype, stats in type_stats.items():
|
||||
valid = stats["correct"] + stats["incorrect"]
|
||||
by_type[qtype] = {
|
||||
**stats,
|
||||
"accuracy": stats["correct"] / stats["total"] if stats["total"] > 0 else 0,
|
||||
"accuracy_valid": stats["correct"] / valid if valid > 0 else 0,
|
||||
}
|
||||
|
||||
return {
|
||||
"overall": overall,
|
||||
"by_question_type": by_type,
|
||||
}
|
||||
|
||||
|
||||
def compute_timing_stats(results: list[dict]) -> dict[str, Any]:
|
||||
"""Compute timing statistics.
|
||||
|
||||
Args:
|
||||
results: List of result dictionaries
|
||||
|
||||
Returns:
|
||||
Dictionary with timing statistics
|
||||
"""
|
||||
summary_times = []
|
||||
retrieve_times = []
|
||||
|
||||
for r in results:
|
||||
summary_ms = r.get("summary_duration_ms", 0)
|
||||
retrieve_ms = r.get("retrieve_duration_ms", 0)
|
||||
|
||||
if summary_ms > 0:
|
||||
summary_times.append(summary_ms)
|
||||
if retrieve_ms > 0:
|
||||
retrieve_times.append(retrieve_ms)
|
||||
|
||||
def compute_stats(times: list[float]) -> dict:
|
||||
if not times:
|
||||
return {"count": 0, "total_ms": 0, "avg_ms": 0, "min_ms": 0, "max_ms": 0}
|
||||
|
||||
return {
|
||||
"count": len(times),
|
||||
"total_ms": sum(times),
|
||||
"total_min": sum(times) / 1000 / 60,
|
||||
"avg_ms": sum(times) / len(times),
|
||||
"min_ms": min(times),
|
||||
"max_ms": max(times),
|
||||
}
|
||||
|
||||
return {
|
||||
"summary": compute_stats(summary_times),
|
||||
"retrieve": compute_stats(retrieve_times),
|
||||
"total_time_min": (sum(summary_times) + sum(retrieve_times)) / 1000 / 60,
|
||||
}
|
||||
|
||||
|
||||
def compute_memory_stats(results: list[dict]) -> dict[str, Any]:
|
||||
"""Compute memory extraction statistics.
|
||||
|
||||
Args:
|
||||
results: List of result dictionaries
|
||||
|
||||
Returns:
|
||||
Dictionary with memory statistics
|
||||
"""
|
||||
memory_counts = []
|
||||
session_counts = []
|
||||
|
||||
for r in results:
|
||||
memories = r.get("extracted_memories", [])
|
||||
num_sessions = r.get("num_sessions", 0)
|
||||
|
||||
memory_counts.append(len(memories))
|
||||
session_counts.append(num_sessions)
|
||||
|
||||
def compute_stats(counts: list[int]) -> dict:
|
||||
if not counts:
|
||||
return {"count": 0, "total": 0, "avg": 0, "min": 0, "max": 0}
|
||||
|
||||
return {
|
||||
"count": len(counts),
|
||||
"total": sum(counts),
|
||||
"avg": sum(counts) / len(counts),
|
||||
"min": min(counts),
|
||||
"max": max(counts),
|
||||
}
|
||||
|
||||
return {
|
||||
"memories_per_question": compute_stats(memory_counts),
|
||||
"sessions_per_question": compute_stats(session_counts),
|
||||
}
|
||||
|
||||
|
||||
def print_report(
|
||||
accuracy_stats: dict,
|
||||
timing_stats: dict,
|
||||
memory_stats: dict,
|
||||
results_dir: str,
|
||||
):
|
||||
"""Print formatted statistics report.
|
||||
|
||||
Args:
|
||||
accuracy_stats: Accuracy statistics
|
||||
timing_stats: Timing statistics
|
||||
memory_stats: Memory statistics
|
||||
results_dir: Path to results directory
|
||||
"""
|
||||
print("\n" + "=" * 80)
|
||||
print("LONGMEMEVAL EVALUATION STATISTICS")
|
||||
print(f"Results Directory: {results_dir}")
|
||||
print("=" * 80)
|
||||
|
||||
# Overall accuracy
|
||||
overall = accuracy_stats["overall"]
|
||||
print("\n📊 Overall Accuracy:")
|
||||
print(f" Total Questions: {overall['total']}")
|
||||
print(f" ✅ Correct: {overall['correct']} ({100 * overall['accuracy']:.2f}%)")
|
||||
print(
|
||||
f" ❌ Incorrect: {overall['incorrect']} "
|
||||
f"({100 * overall['incorrect'] / overall['total'] if overall['total'] > 0 else 0:.2f}%)",
|
||||
)
|
||||
if overall["error"] > 0:
|
||||
print(f" ⚠️ Error: {overall['error']} ({100 * overall['error'] / overall['total']:.2f}%)")
|
||||
print(f" Accuracy (valid): {100 * overall['accuracy_valid']:.2f}%")
|
||||
|
||||
# Accuracy by question type
|
||||
print("\n📊 Accuracy by Question Type:")
|
||||
print("-" * 60)
|
||||
print(f"{'Question Type':<30} {'Correct':<10} {'Total':<10} {'Accuracy':<10}")
|
||||
print("-" * 60)
|
||||
|
||||
by_type = accuracy_stats["by_question_type"]
|
||||
for qtype in sorted(by_type.keys()):
|
||||
stats = by_type[qtype]
|
||||
print(f"{qtype:<30} {stats['correct']:<10} {stats['total']:<10} {100 * stats['accuracy']:.2f}%")
|
||||
|
||||
print("-" * 60)
|
||||
|
||||
# Timing statistics
|
||||
print("\n⏱️ Timing Statistics:")
|
||||
summary = timing_stats["summary"]
|
||||
retrieve = timing_stats["retrieve"]
|
||||
|
||||
print(" Memory Summarization:")
|
||||
print(f" Total Time: {summary['total_min']:.2f} min")
|
||||
print(f" Avg per Q: {summary['avg_ms']:.0f} ms")
|
||||
print(f" Min/Max: {summary['min_ms']:.0f} / {summary['max_ms']:.0f} ms")
|
||||
|
||||
print(" Memory Retrieval:")
|
||||
print(f" Total Time: {retrieve['total_min']:.2f} min")
|
||||
print(f" Avg per Q: {retrieve['avg_ms']:.0f} ms")
|
||||
print(f" Min/Max: {retrieve['min_ms']:.0f} / {retrieve['max_ms']:.0f} ms")
|
||||
|
||||
print(f" Total Time: {timing_stats['total_time_min']:.2f} min")
|
||||
|
||||
# Memory statistics
|
||||
print("\n📝 Memory Statistics:")
|
||||
mem = memory_stats["memories_per_question"]
|
||||
sess = memory_stats["sessions_per_question"]
|
||||
|
||||
print(" Extracted Memories per Question:")
|
||||
print(f" Total: {mem['total']}")
|
||||
print(f" Average: {mem['avg']:.1f}")
|
||||
print(f" Min/Max: {mem['min']} / {mem['max']}")
|
||||
|
||||
print(" Sessions per Question:")
|
||||
print(f" Average: {sess['avg']:.1f}")
|
||||
print(f" Min/Max: {sess['min']} / {sess['max']}")
|
||||
|
||||
print("\n" + "=" * 80)
|
||||
|
||||
|
||||
def save_statistics(
|
||||
accuracy_stats: dict,
|
||||
timing_stats: dict,
|
||||
memory_stats: dict,
|
||||
output_file: str,
|
||||
):
|
||||
"""Save statistics to JSON file.
|
||||
|
||||
Args:
|
||||
accuracy_stats: Accuracy statistics
|
||||
timing_stats: Timing statistics
|
||||
memory_stats: Memory statistics
|
||||
output_file: Path to output file
|
||||
"""
|
||||
stats = {
|
||||
"accuracy": accuracy_stats,
|
||||
"timing": timing_stats,
|
||||
"memory": memory_stats,
|
||||
}
|
||||
|
||||
with open(output_file, "w", encoding="utf-8") as f:
|
||||
json.dump(stats, f, indent=4, ensure_ascii=False)
|
||||
|
||||
print(f"\n📁 Statistics saved to: {output_file}")
|
||||
|
||||
|
||||
def main(results_dir: str, output_file: str = None):
|
||||
"""Main function to compute and display statistics.
|
||||
|
||||
Args:
|
||||
results_dir: Path to results directory
|
||||
output_file: Optional path to save statistics JSON
|
||||
"""
|
||||
print(f"\nLoading results from: {results_dir}")
|
||||
|
||||
results = load_results(results_dir)
|
||||
|
||||
if not results:
|
||||
print("❌ No results found!")
|
||||
return
|
||||
|
||||
print(f"Loaded {len(results)} question results")
|
||||
|
||||
# Compute statistics
|
||||
accuracy_stats = compute_accuracy_stats(results)
|
||||
timing_stats = compute_timing_stats(results)
|
||||
memory_stats = compute_memory_stats(results)
|
||||
|
||||
# Print report
|
||||
print_report(accuracy_stats, timing_stats, memory_stats, results_dir)
|
||||
|
||||
# Save to file if specified
|
||||
if output_file:
|
||||
save_statistics(accuracy_stats, timing_stats, memory_stats, output_file)
|
||||
else:
|
||||
# Default output file in results directory
|
||||
default_output = Path(results_dir) / "statistics.json"
|
||||
save_statistics(accuracy_stats, timing_stats, memory_stats, str(default_output))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Compute statistics from LongMemEval evaluation results",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--results_dir",
|
||||
type=str,
|
||||
default="bench_results/longmemeval_reme",
|
||||
help="Path to results directory containing question_*.json files",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output_file",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Path to save statistics JSON (default: <results_dir>/statistics.json)",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
main(
|
||||
results_dir=args.results_dir,
|
||||
output_file=args.output_file,
|
||||
)
|
||||
1040
benchmark/longmemeval/eval_longmemeval_reme.py
Normal file
1040
benchmark/longmemeval/eval_longmemeval_reme.py
Normal file
File diff suppressed because it is too large
Load diff
921
benchmark/longmemeval/eval_longmemeval_reme_retrieve.py
Normal file
921
benchmark/longmemeval/eval_longmemeval_reme_retrieve.py
Normal file
|
|
@ -0,0 +1,921 @@
|
|||
"""
|
||||
LongMemEval Benchmark Evaluator for ReMe - Retrieve Only
|
||||
|
||||
A simplified evaluation pipeline that only runs the retrieve and judge phases:
|
||||
1. Loads LongMemEval benchmark data
|
||||
2. Skips memory summarization (assumes memories are already in vector store)
|
||||
3. Uses questions to query memory and generate answers
|
||||
4. Uses LLM to judge answer correctness
|
||||
5. Generates comprehensive metrics
|
||||
|
||||
This is useful for debugging/tuning the retrieve phase without re-running summary.
|
||||
|
||||
Usage:
|
||||
python benchmark/longmemeval/eval_longmemeval_reme_retrieve.py \
|
||||
--data_path dataset/longmemeval/longmemeval_s_cleaned.json \
|
||||
--top_k 20 --start_index 0 --end_index 10
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from reme.reme import ReMe
|
||||
|
||||
|
||||
# ==================== Configuration ====================
|
||||
|
||||
|
||||
@dataclass
|
||||
class RetrieveEvalConfig:
|
||||
"""Evaluation configuration parameters for retrieve-only mode."""
|
||||
|
||||
data_path: str
|
||||
top_k: int = 10
|
||||
start_index: int = 0
|
||||
end_index: Optional[int] = None
|
||||
max_concurrency: int = 1
|
||||
output_dir: str = "cache/bench_results/longmemeval_reme_retrieve"
|
||||
reme_model_name: str = "qwen-flash"
|
||||
eval_model_name: str = "qwen3-max"
|
||||
algo_version: str = "v1"
|
||||
samples_per_type: int = -1 # Number of samples per question type, -1 for all
|
||||
enable_thinking_params: bool = False
|
||||
# Optional: path to previous summary results to reload memories
|
||||
summary_results_dir: Optional[str] = None
|
||||
|
||||
|
||||
# ==================== Answer Judge Prompts ====================
|
||||
|
||||
|
||||
def get_anscheck_prompt(task: str, question: str, answer: str, response: str, abstention: bool = False) -> str:
|
||||
"""Generate the answer checking prompt based on question type.
|
||||
|
||||
Args:
|
||||
task: Question type, e.g. 'single-session-user', 'multi-session', 'temporal-reasoning'
|
||||
question: The question content
|
||||
answer: The reference answer
|
||||
response: The model's response
|
||||
abstention: Whether this is an unanswerable question
|
||||
|
||||
Returns:
|
||||
Prompt for judging answer correctness
|
||||
"""
|
||||
if not abstention:
|
||||
if task in ["single-session-user", "single-session-assistant", "multi-session"]:
|
||||
template = (
|
||||
"I will give you a question, a correct answer, and a response from a model. Please answer yes i"
|
||||
"f the response contains the correct answer. Otherwise, answer no. If the response is equival"
|
||||
"ent to the correct answer or contains all the intermediate steps to get the correct answer, "
|
||||
"you should also answer yes. If the response only contains a subset of the information required"
|
||||
" by the answer, answer no. \n\nQuestion: {}\n\nCorrect Answer: {}\n\nModel Response: {}\n\nIs"
|
||||
" the model response correct? Answer yes or no only."
|
||||
)
|
||||
prompt = template.format(question, answer, response)
|
||||
elif task == "temporal-reasoning":
|
||||
template = (
|
||||
"I will give you a question, a correct answer, and a response from a model. Please answer yes"
|
||||
" if the response contains the correct answer. Otherwise, answer no. If the response is equiv"
|
||||
"alent to the correct answer or contains all the intermediate steps to get the correct answer"
|
||||
", you should also answer yes. If the response only contains a subset of the information requ"
|
||||
"ired by the answer, answer no. In addition, do not penalize off-by-one errors for the numbe"
|
||||
"r of days. If the question asks for the number of days/weeks/months, etc., and the model ma"
|
||||
"kes off-by-one errors (e.g., predicting 19 days when the answer is 18), the model's respon"
|
||||
"se is still correct. \n\nQuestion: {}\n\nCorrect Answer: {}\n\nModel Response: {}\n\nIs th"
|
||||
"e model response correct? Answer yes or no only."
|
||||
)
|
||||
prompt = template.format(question, answer, response)
|
||||
elif task == "knowledge-update":
|
||||
template = (
|
||||
"I will give you a question, a correct answer, and a response from a model. Please answer yes "
|
||||
"if the response contains the correct answer. Otherwise, answer no. If the response contains "
|
||||
"some previous information along with an updated answer, the response should be considered "
|
||||
"as correct as long as the updated answer is the required answer.\n\nQuestion: {}\n\nCorrec"
|
||||
"t Answer: {}\n\nModel Response: {}\n\nIs the model response correct? Answer yes or no only."
|
||||
)
|
||||
prompt = template.format(question, answer, response)
|
||||
elif task == "single-session-preference":
|
||||
template = (
|
||||
"I will give you a question, a rubric for desired personalized response, and a response from a"
|
||||
" model. Please answer yes if the response satisfies the desired response. Otherwise, answer"
|
||||
" no. The model does not need to reflect all the points in the rubric. The response is corr"
|
||||
"ect as long as it recalls and utilizes the user's personal information correctly.\n\nQuest"
|
||||
"ion: {}\n\nRubric: {}\n\nModel Response: {}\n\nIs the model response correct? Answer yes o"
|
||||
"r no only."
|
||||
)
|
||||
prompt = template.format(question, answer, response)
|
||||
else:
|
||||
# Default template
|
||||
template = (
|
||||
"I will give you a question, a correct answer, and a response from a model. Please answer y"
|
||||
"es if the response contains the correct answer. Otherwise, answer no. If the response is "
|
||||
"equivalent to the correct answer or contains all the intermediate steps to get the correc"
|
||||
"t answer, you should also answer yes. If the response only contains a subset of the infor"
|
||||
"mation required by the answer, answer no. \n\nQuestion: {}\n\nCorrect Answer: {}\n\nModel"
|
||||
" Response: {}\n\nIs the model response correct? Answer yes or no only."
|
||||
)
|
||||
prompt = template.format(question, answer, response)
|
||||
|
||||
else:
|
||||
template = (
|
||||
"I will give you an unanswerable question, an explanation, and a response from a model. Please "
|
||||
"answer yes if the model correctly identifies the question as unanswerable. The model could say "
|
||||
"that the information is incomplete, or some other information is given but the asked informati"
|
||||
"on is not.\n\nQuestion: {}\n\nExplanation: {}\n\nModel Response: {}\n\nDoes the model correct"
|
||||
"ly identify the question as unanswerable? Answer yes or no only."
|
||||
)
|
||||
prompt = template.format(question, answer, response)
|
||||
return prompt
|
||||
|
||||
|
||||
# ==================== Utilities ====================
|
||||
|
||||
|
||||
class DataLoader:
|
||||
"""Handles loading and parsing of LongMemEval data."""
|
||||
|
||||
@staticmethod
|
||||
def load_json(file_path: str) -> list[dict]:
|
||||
"""Load all entries from a JSON file."""
|
||||
with open(file_path, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
|
||||
@staticmethod
|
||||
def filter_by_type(data: list[dict], samples_per_type: int = -1) -> list[tuple[int, dict]]:
|
||||
"""Filter data by question type with specified number of samples per type.
|
||||
|
||||
Args:
|
||||
data: List of question entries
|
||||
samples_per_type: Number of samples per type, -1 for all
|
||||
|
||||
Returns:
|
||||
List of tuples (original_index, entry) for selected samples
|
||||
"""
|
||||
if samples_per_type == -1:
|
||||
# Return all with original indices
|
||||
return list(enumerate(data))
|
||||
|
||||
# Group by question type
|
||||
type_groups: dict[str, list[tuple[int, dict]]] = {}
|
||||
for i, entry in enumerate(data):
|
||||
qtype = entry.get("question_type", "unknown")
|
||||
if qtype not in type_groups:
|
||||
type_groups[qtype] = []
|
||||
type_groups[qtype].append((i, entry))
|
||||
|
||||
# Select samples from each type
|
||||
selected = []
|
||||
for qtype, entries in type_groups.items():
|
||||
count = min(samples_per_type, len(entries))
|
||||
selected.extend(entries[:count])
|
||||
logger.info(f" {qtype}: selected {count}/{len(entries)} samples")
|
||||
|
||||
# Sort by original index to maintain order
|
||||
selected.sort(key=lambda x: x[0])
|
||||
return selected
|
||||
|
||||
|
||||
class FileManager:
|
||||
"""Manages file I/O operations."""
|
||||
|
||||
def __init__(self, base_dir: str):
|
||||
self.base_dir = Path(base_dir)
|
||||
self.base_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def save_question_result(self, idx: int, question_id: str, data: dict):
|
||||
"""Save result for a single question."""
|
||||
file_path = self.base_dir / f"question_{idx:04d}_{question_id}.json"
|
||||
with open(file_path, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=4, ensure_ascii=False)
|
||||
logger.info(f"✅ Saved question result to {file_path}")
|
||||
|
||||
def load_question_result(self, idx: int, question_id: str) -> Optional[dict]:
|
||||
"""Load result for a single question if exists."""
|
||||
file_path = self.base_dir / f"question_{idx:04d}_{question_id}.json"
|
||||
if not file_path.exists():
|
||||
return None
|
||||
with open(file_path, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
|
||||
def save_summary(self, results: list[dict]):
|
||||
"""Save summary of all results."""
|
||||
file_path = self.base_dir / "summary.json"
|
||||
with open(file_path, "w", encoding="utf-8") as f:
|
||||
json.dump(results, f, indent=4, ensure_ascii=False)
|
||||
logger.info(f"✅ Saved summary to {file_path}")
|
||||
|
||||
|
||||
# ==================== Evaluation Functions ====================
|
||||
|
||||
|
||||
async def answer_question_with_memories(
|
||||
reme: ReMe,
|
||||
question: str,
|
||||
memories: str,
|
||||
user_id: str = None,
|
||||
model_name: str = "qwen3-max",
|
||||
):
|
||||
"""
|
||||
Answer a question using retrieved memories with PROMPT_MEMZERO_JSON template.
|
||||
|
||||
Args:
|
||||
reme: ReMe instance with default_llm and prompt_handler
|
||||
question: The question to answer
|
||||
memories: The retrieved memories (formatted as context)
|
||||
user_id: Optional user ID for context formatting
|
||||
model_name: Model name to use for LLM request
|
||||
|
||||
Returns:
|
||||
dict with 'reasoning' and 'answer' fields
|
||||
"""
|
||||
# Format context with memories
|
||||
if user_id:
|
||||
context = reme.prompt_handler.prompt_format(
|
||||
"TEMPLATE_MEMOS",
|
||||
user_id=user_id,
|
||||
memories=memories,
|
||||
)
|
||||
else:
|
||||
context = f"Memories:\n{memories}"
|
||||
|
||||
# Use PROMPT_MEMZERO_JSON template for structured JSON response
|
||||
prompt = reme.prompt_handler.prompt_format(
|
||||
"PROMPT_MEMZERO_JSON",
|
||||
context=context,
|
||||
question=question,
|
||||
)
|
||||
|
||||
result = await reme.get_llm(name=model_name).simple_request_for_json(
|
||||
prompt=prompt,
|
||||
model_name=None,
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# ==================== Memory Operations ====================
|
||||
|
||||
|
||||
class RetrieveProcessor:
|
||||
"""Handles ReMe memory retrieve operations only."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
reme: ReMe,
|
||||
reme_model_name: str = "qwen-flash",
|
||||
eval_model_name: str = "qwen3-max",
|
||||
algo_version: str = "v1",
|
||||
enable_thinking_params: bool = False,
|
||||
):
|
||||
self.reme = reme
|
||||
self.reme_model_name = reme_model_name
|
||||
self.eval_model_name = eval_model_name
|
||||
self.algo_version = algo_version
|
||||
self.enable_thinking_params = enable_thinking_params
|
||||
|
||||
async def search_memory(
|
||||
self,
|
||||
query: str,
|
||||
user_id: str,
|
||||
top_k: int = 20,
|
||||
) -> tuple[dict, list, float]:
|
||||
"""
|
||||
Search memory using ReMe and return structured answer with reasoning.
|
||||
|
||||
Returns:
|
||||
tuple: (answer_dict, agent_messages, duration_ms)
|
||||
answer_dict contains: {"reasoning": str, "answer": str, "memories": str}
|
||||
"""
|
||||
start = time.time()
|
||||
|
||||
# Retrieve memories from ReMe using new API
|
||||
result = await self.reme.retrieve_memory(
|
||||
llm_config_name="qwen3-max",
|
||||
query=query,
|
||||
retrieve_top_k=top_k,
|
||||
user_name=user_id,
|
||||
version=self.algo_version,
|
||||
return_dict=True,
|
||||
enable_time_filter=True,
|
||||
enable_thinking_params=True,
|
||||
)
|
||||
|
||||
# Extract memories from response
|
||||
memories = result["answer"]
|
||||
agent_messages = [x.simple_dump(enable_argument_dict=True) for x in result["messages"]]
|
||||
retrieved_nodes = [x.model_dump(exclude_none=True) for x in result["retrieved_nodes"]]
|
||||
|
||||
# Use LLM to generate structured answer from memories
|
||||
answer_result = await answer_question_with_memories(
|
||||
reme=self.reme,
|
||||
question=query,
|
||||
memories=memories,
|
||||
user_id=user_id,
|
||||
model_name=self.eval_model_name,
|
||||
)
|
||||
|
||||
# Add original memories to the result
|
||||
answer_result["memories"] = memories
|
||||
answer_result["retrieved_nodes"] = retrieved_nodes
|
||||
|
||||
duration_ms = (time.time() - start) * 1000
|
||||
return answer_result, agent_messages, duration_ms
|
||||
|
||||
|
||||
# ==================== Answer Judge ====================
|
||||
|
||||
|
||||
class LongMemEvalJudge:
|
||||
"""LongMemEval answer judge using LLM."""
|
||||
|
||||
def __init__(self, reme: ReMe, model: str = "qwen3-max"):
|
||||
self.reme = reme
|
||||
self.model = model
|
||||
|
||||
async def judge_answer(
|
||||
self,
|
||||
question_type: str,
|
||||
question: str,
|
||||
answer: str,
|
||||
response: str,
|
||||
abstention: bool = False,
|
||||
) -> dict:
|
||||
"""
|
||||
Judge if the model's response is correct.
|
||||
|
||||
Returns:
|
||||
dict with is_correct, llm_response, and judge_prompt
|
||||
"""
|
||||
prompt = get_anscheck_prompt(question_type, question, answer, response, abstention)
|
||||
|
||||
try:
|
||||
llm_response = await self.reme.get_llm("default").simple_request(
|
||||
prompt=prompt,
|
||||
model_name=self.model,
|
||||
)
|
||||
llm_response_lower = llm_response.strip().lower()
|
||||
is_correct = llm_response_lower.startswith("yes")
|
||||
|
||||
return {
|
||||
"is_correct": is_correct,
|
||||
"llm_response": llm_response,
|
||||
"judge_prompt": prompt,
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"is_correct": None,
|
||||
"error": str(e),
|
||||
"judge_prompt": prompt,
|
||||
}
|
||||
|
||||
|
||||
# ==================== Metrics ====================
|
||||
|
||||
|
||||
class MetricsAggregator:
|
||||
"""Aggregates evaluation metrics for LongMemEval."""
|
||||
|
||||
@staticmethod
|
||||
def compute_metrics(results: list[dict]) -> dict[str, Any]:
|
||||
"""Compute overall and per-type metrics."""
|
||||
total = len(results)
|
||||
correct = sum(1 for r in results if r.get("judgment", {}).get("is_correct") is True)
|
||||
incorrect = sum(1 for r in results if r.get("judgment", {}).get("is_correct") is False)
|
||||
error = total - correct - incorrect
|
||||
|
||||
metrics = {
|
||||
"total": total,
|
||||
"correct": correct,
|
||||
"incorrect": incorrect,
|
||||
"error": error,
|
||||
"accuracy": correct / total if total > 0 else 0,
|
||||
"accuracy_valid": correct / (correct + incorrect) if (correct + incorrect) > 0 else 0,
|
||||
}
|
||||
|
||||
# Per question type statistics
|
||||
type_stats = {}
|
||||
for r in results:
|
||||
qtype = r.get("question_type", "unknown")
|
||||
if qtype not in type_stats:
|
||||
type_stats[qtype] = {"total": 0, "correct": 0, "incorrect": 0}
|
||||
type_stats[qtype]["total"] += 1
|
||||
if r.get("judgment", {}).get("is_correct") is True:
|
||||
type_stats[qtype]["correct"] += 1
|
||||
elif r.get("judgment", {}).get("is_correct") is False:
|
||||
type_stats[qtype]["incorrect"] += 1
|
||||
|
||||
metrics["by_question_type"] = {
|
||||
qtype: {
|
||||
**stats,
|
||||
"accuracy": stats["correct"] / stats["total"] if stats["total"] > 0 else 0,
|
||||
"accuracy_valid": (
|
||||
stats["correct"] / (stats["correct"] + stats["incorrect"])
|
||||
if (stats["correct"] + stats["incorrect"]) > 0
|
||||
else 0
|
||||
),
|
||||
}
|
||||
for qtype, stats in type_stats.items()
|
||||
}
|
||||
|
||||
return metrics
|
||||
|
||||
@staticmethod
|
||||
def compute_timing_stats(results: list[dict]) -> dict[str, Any]:
|
||||
"""Compute timing statistics."""
|
||||
retrieve_times = []
|
||||
|
||||
for r in results:
|
||||
retrieve_ms = r.get("retrieve_duration_ms", 0)
|
||||
if retrieve_ms > 0:
|
||||
retrieve_times.append(retrieve_ms)
|
||||
|
||||
def compute_stats(times: list[float]) -> dict:
|
||||
if not times:
|
||||
return {"count": 0, "total_ms": 0, "avg_ms": 0, "min_ms": 0, "max_ms": 0}
|
||||
|
||||
return {
|
||||
"count": len(times),
|
||||
"total_ms": sum(times),
|
||||
"total_min": sum(times) / 1000 / 60,
|
||||
"avg_ms": sum(times) / len(times),
|
||||
"min_ms": min(times),
|
||||
"max_ms": max(times),
|
||||
}
|
||||
|
||||
return {
|
||||
"retrieve": compute_stats(retrieve_times),
|
||||
"total_time_min": sum(retrieve_times) / 1000 / 60,
|
||||
}
|
||||
|
||||
|
||||
# ==================== Main Pipeline ====================
|
||||
|
||||
|
||||
class LongMemEvalRetrieveEvaluator:
|
||||
"""Retrieve-only evaluator for LongMemEval benchmark using ReMe."""
|
||||
|
||||
def __init__(self, config: RetrieveEvalConfig):
|
||||
self.config = config
|
||||
self.reme = ReMe(
|
||||
default_llm_config={
|
||||
"model_name": self.config.reme_model_name,
|
||||
},
|
||||
llms={
|
||||
"qwen-plus-think": {
|
||||
"backend": "openai",
|
||||
"model_name": "qwen-plus",
|
||||
"extra_body": {
|
||||
"enable_thinking": True,
|
||||
},
|
||||
},
|
||||
"qwen3-max-think": {
|
||||
"backend": "openai",
|
||||
"model_name": "qwen3-max",
|
||||
"extra_body": {
|
||||
"enable_thinking": True,
|
||||
},
|
||||
},
|
||||
"qwen3-max": {
|
||||
"backend": "openai",
|
||||
"model_name": "qwen3-max",
|
||||
"extra_body": {
|
||||
"enable_thinking": False,
|
||||
},
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
# Load evaluation prompts into ReMe's prompt handler
|
||||
prompts_yaml_path = Path(__file__).parent / "eval_reme.yaml"
|
||||
self.reme.prompt_handler.load_prompt_by_file(prompts_yaml_path)
|
||||
|
||||
self.file_manager = FileManager(config.output_dir)
|
||||
self.retrieve_processor = RetrieveProcessor(
|
||||
self.reme,
|
||||
config.reme_model_name,
|
||||
config.eval_model_name,
|
||||
config.algo_version,
|
||||
config.enable_thinking_params,
|
||||
)
|
||||
self.judge = LongMemEvalJudge(self.reme, config.eval_model_name)
|
||||
self.data_loader = DataLoader()
|
||||
|
||||
async def __aenter__(self):
|
||||
"""Async context manager entry."""
|
||||
await self.reme.start()
|
||||
return self
|
||||
|
||||
async def __aexit__(self, exc_type, exc_val, exc_tb):
|
||||
"""Async context manager exit with cleanup."""
|
||||
await self.reme.close()
|
||||
return False
|
||||
|
||||
async def process_question_entry(self, entry: dict, idx: int) -> dict:
|
||||
"""Process a single question entry (retrieve + judge only).
|
||||
|
||||
Args:
|
||||
entry: A question entry from LongMemEval dataset
|
||||
idx: Index of the question
|
||||
|
||||
Returns:
|
||||
Result dictionary
|
||||
"""
|
||||
question_id = entry["question_id"]
|
||||
question = entry["question"]
|
||||
answer = entry["answer"]
|
||||
question_type = entry["question_type"]
|
||||
question_date = entry.get("question_date", "")
|
||||
haystack_dates = entry["haystack_dates"]
|
||||
haystack_session_ids = entry["haystack_session_ids"]
|
||||
haystack_sessions = entry["haystack_sessions"]
|
||||
|
||||
# Use question_id as user_id for isolation (same as full eval)
|
||||
user_id = f"longmemeval_{question_id}"
|
||||
|
||||
logger.info(f"\n{'='*60}")
|
||||
logger.info(f"Question ID: {question_id}")
|
||||
logger.info(f"Question Type: {question_type}")
|
||||
logger.info(f"Question: {question}")
|
||||
logger.info(f"Question_date: {question_date}")
|
||||
logger.info(f"Answer: {answer}")
|
||||
logger.info(f"Number of sessions: {len(haystack_sessions)}")
|
||||
logger.info(f"{'='*60}")
|
||||
|
||||
# Skip summary phase - directly search memory and answer question
|
||||
logger.info(" Retrieving and answering question using ReMe...")
|
||||
answer_dict, retrieve_messages, retrieve_duration_ms = await self.retrieve_processor.search_memory(
|
||||
query=f"[Question_date: {question_date} | Question_type: {question_type}] " + question,
|
||||
user_id=user_id,
|
||||
top_k=self.config.top_k,
|
||||
)
|
||||
|
||||
# Extract answer and reasoning from the structured response
|
||||
model_response = answer_dict.get("answer", "")
|
||||
model_reasoning = answer_dict.get("reasoning", "")
|
||||
retrieved_memories = answer_dict.get("memories", "")
|
||||
retrieved_nodes = answer_dict.get("retrieved_nodes", [])
|
||||
|
||||
# Judge answer correctness
|
||||
logger.info(" Judging answer correctness...")
|
||||
judgment = await self.judge.judge_answer(
|
||||
question_type=question_type,
|
||||
question=question,
|
||||
answer=answer,
|
||||
response=model_response,
|
||||
)
|
||||
|
||||
is_correct = judgment.get("is_correct")
|
||||
logger.info(
|
||||
f" → Answer judgment: {'Correct' if is_correct else 'Incorrect' if is_correct is False else 'Error'}",
|
||||
)
|
||||
|
||||
result = {
|
||||
"question_id": question_id,
|
||||
"question_type": question_type,
|
||||
"question": question,
|
||||
"answer": answer,
|
||||
"question_date": question_date,
|
||||
"haystack_dates": haystack_dates,
|
||||
"haystack_session_ids": haystack_session_ids,
|
||||
"num_sessions": len(haystack_sessions),
|
||||
"model_response": model_response,
|
||||
"model_reasoning": model_reasoning,
|
||||
"retrieved_memories": retrieved_memories,
|
||||
"retrieved_nodes": retrieved_nodes,
|
||||
"judgment": judgment,
|
||||
"retrieve_duration_ms": retrieve_duration_ms,
|
||||
"retrieve_messages": retrieve_messages,
|
||||
}
|
||||
|
||||
# Save individual result
|
||||
self.file_manager.save_question_result(idx, question_id, result)
|
||||
|
||||
logger.info(f" Question {question_id} - Completed")
|
||||
|
||||
return result
|
||||
|
||||
async def run_evaluation(self):
|
||||
"""Run the retrieve-only evaluation pipeline with parallel processing."""
|
||||
start_time = time.time()
|
||||
|
||||
# NOTE: Do NOT clear vector store - we assume memories are already there from previous summary run
|
||||
|
||||
# Load dataset
|
||||
logger.info(f"Loading dataset from: {self.config.data_path}")
|
||||
all_data = self.data_loader.load_json(self.config.data_path)
|
||||
logger.info(f"Total questions in dataset: {len(all_data)}")
|
||||
|
||||
# Filter by question type
|
||||
logger.info(f"Filtering by type (samples_per_type={self.config.samples_per_type}):")
|
||||
filtered_data = self.data_loader.filter_by_type(all_data, self.config.samples_per_type)
|
||||
logger.info(f"Selected {len(filtered_data)} questions after filtering")
|
||||
|
||||
# Apply start_index and end_index on filtered data
|
||||
end_index = self.config.end_index or len(filtered_data)
|
||||
start_index = self.config.start_index
|
||||
end_index = min(end_index, len(filtered_data))
|
||||
|
||||
# Get the slice we want to process
|
||||
data_to_process = filtered_data[start_index:end_index]
|
||||
total_questions = len(data_to_process)
|
||||
|
||||
logger.info(f"Processing {total_questions} questions (index {start_index} to {end_index - 1})")
|
||||
|
||||
print("\n" + "=" * 80)
|
||||
print("LONGMEMEVAL EVALUATION - REME (RETRIEVE ONLY)")
|
||||
print(f"Samples per type: {self.config.samples_per_type} (-1 = all)")
|
||||
print(f"Questions to process: {total_questions} | Top-K: {self.config.top_k}")
|
||||
print(f"Max Concurrency: {self.config.max_concurrency}")
|
||||
print(f"ReMe Model: {self.config.reme_model_name} | Eval Model: {self.config.eval_model_name}")
|
||||
print(f"Algo Version: {self.config.algo_version}")
|
||||
print("⚠️ NOTE: Assumes memories are already in vector store from previous summary run")
|
||||
print("=" * 80 + "\n")
|
||||
|
||||
# Use semaphore to control concurrency
|
||||
semaphore = asyncio.Semaphore(self.config.max_concurrency)
|
||||
|
||||
async def process_with_semaphore(idx: int, original_idx: int, entry: dict) -> Optional[dict]:
|
||||
"""Process a question with semaphore for concurrency control."""
|
||||
async with semaphore:
|
||||
question_id = entry["question_id"]
|
||||
|
||||
# Check cache first (use original index for cache file naming)
|
||||
cached_result = self.file_manager.load_question_result(original_idx, question_id)
|
||||
if cached_result:
|
||||
print(f"⚡ [{idx}/{total_questions}] Skipping question {original_idx} (cached)")
|
||||
return cached_result
|
||||
|
||||
print(f"\n{'#'*60}")
|
||||
print(f"### [{idx}/{total_questions}] Processing Question {original_idx} ###")
|
||||
print(f"{'#'*60}")
|
||||
|
||||
try:
|
||||
result = await self.process_question_entry(entry, original_idx)
|
||||
print(f"✅ [{idx}/{total_questions}] Completed question {original_idx}")
|
||||
return result
|
||||
except Exception as e:
|
||||
logger.error(f"❌ Error processing question {original_idx}: {e}")
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
return {
|
||||
"question_id": question_id,
|
||||
"error": str(e),
|
||||
"question_type": entry.get("question_type", "unknown"),
|
||||
"question": entry.get("question", ""),
|
||||
"answer": entry.get("answer", ""),
|
||||
"judgment": {"is_correct": None, "error": str(e)},
|
||||
}
|
||||
|
||||
# Create all tasks from filtered data (each item is a tuple of (original_idx, entry))
|
||||
tasks = [
|
||||
process_with_semaphore(idx + 1, original_idx, entry)
|
||||
for idx, (original_idx, entry) in enumerate(data_to_process)
|
||||
]
|
||||
|
||||
# Execute in parallel with controlled concurrency
|
||||
all_results = await asyncio.gather(*tasks, return_exceptions=False)
|
||||
|
||||
# Filter out None results if any
|
||||
all_results = [r for r in all_results if r is not None]
|
||||
|
||||
# Save summary
|
||||
self.file_manager.save_summary(all_results)
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
print(f"\n✅ Processing completed in {elapsed:.2f}s")
|
||||
if total_questions > 0:
|
||||
print(f" Average time per question: {elapsed / total_questions:.2f}s")
|
||||
|
||||
# Compute and report metrics
|
||||
self._report_metrics(all_results)
|
||||
|
||||
return all_results
|
||||
|
||||
def _report_metrics(self, results: list[dict]):
|
||||
"""Report evaluation metrics."""
|
||||
metrics = MetricsAggregator.compute_metrics(results)
|
||||
timing_stats = MetricsAggregator.compute_timing_stats(results)
|
||||
|
||||
print("\n" + "=" * 80)
|
||||
print("EVALUATION SUMMARY - LONGMEMEVAL - REME (RETRIEVE ONLY)")
|
||||
print("=" * 80 + "\n")
|
||||
|
||||
print("📊 Overall Results:")
|
||||
print(f" ✅ Correct: {metrics['correct']}/{metrics['total']} ({100*metrics['accuracy']:.2f}%)")
|
||||
print(
|
||||
f" ❌ Incorrect: {metrics['incorrect']}/{metrics['total']}"
|
||||
f" ({100*metrics['incorrect']/metrics['total'] if metrics['total'] > 0 else 0:.2f}%)",
|
||||
)
|
||||
if metrics["error"] > 0:
|
||||
print(f" ⚠️ Error: {metrics['error']}/{metrics['total']} ({100*metrics['error']/metrics['total']:.2f}%)")
|
||||
print(f" Accuracy (valid): {100*metrics['accuracy_valid']:.2f}%")
|
||||
|
||||
print("\n📊 Accuracy by Question Type:")
|
||||
print("-" * 60)
|
||||
print(f"{'Question Type':<30} {'Correct':<10} {'Total':<10} {'Accuracy':<10}")
|
||||
print("-" * 60)
|
||||
for qtype in sorted(metrics["by_question_type"].keys()):
|
||||
stats = metrics["by_question_type"][qtype]
|
||||
print(f"{qtype:<30} {stats['correct']:<10} {stats['total']:<10} {100*stats['accuracy']:.2f}%")
|
||||
print("-" * 60)
|
||||
|
||||
print("\n⏱️ Timing Statistics (Retrieve Only):")
|
||||
retrieve = timing_stats["retrieve"]
|
||||
print(" Memory Retrieval:")
|
||||
print(f" Total Time: {retrieve['total_min']:.2f} min")
|
||||
print(f" Avg per Q: {retrieve['avg_ms']:.0f} ms")
|
||||
print(f" Total Time: {timing_stats['total_time_min']:.2f} min")
|
||||
|
||||
# Save metrics
|
||||
final_results = {
|
||||
"accuracy": metrics,
|
||||
"timing": timing_stats,
|
||||
}
|
||||
metrics_file = self.file_manager.base_dir / "eval_statistics.json"
|
||||
with open(metrics_file, "w", encoding="utf-8") as f:
|
||||
json.dump(final_results, f, indent=4, ensure_ascii=False)
|
||||
print(f"\n📁 Statistics saved to: {metrics_file}")
|
||||
|
||||
print("\n" + "=" * 80)
|
||||
|
||||
|
||||
# ==================== Entry Point ====================
|
||||
|
||||
|
||||
async def main_async(
|
||||
data_path: str,
|
||||
top_k: int = 20,
|
||||
start_index: int = 0,
|
||||
end_index: Optional[int] = None,
|
||||
max_concurrency: int = 1,
|
||||
output_dir: str = "bench_results/longmemeval_reme_retrieve",
|
||||
reme_model_name: str = "qwen-flash",
|
||||
eval_model_name: str = "qwen3-max",
|
||||
algo_version: str = "v1",
|
||||
samples_per_type: int = -1,
|
||||
enable_thinking_params: bool = False,
|
||||
summary_results_dir: Optional[str] = None,
|
||||
):
|
||||
"""Main async entry point for LongMemEval retrieve-only evaluation with proper resource cleanup."""
|
||||
config = RetrieveEvalConfig(
|
||||
data_path=data_path,
|
||||
top_k=top_k,
|
||||
start_index=start_index,
|
||||
end_index=end_index,
|
||||
max_concurrency=max_concurrency,
|
||||
output_dir=output_dir,
|
||||
reme_model_name=reme_model_name,
|
||||
eval_model_name=eval_model_name,
|
||||
algo_version=algo_version,
|
||||
samples_per_type=samples_per_type,
|
||||
enable_thinking_params=enable_thinking_params,
|
||||
summary_results_dir=summary_results_dir,
|
||||
)
|
||||
|
||||
# Use async context manager for automatic cleanup
|
||||
async with LongMemEvalRetrieveEvaluator(config) as evaluator:
|
||||
await evaluator.run_evaluation()
|
||||
|
||||
|
||||
def main(
|
||||
data_path: str,
|
||||
top_k: int = 20,
|
||||
start_index: int = 0,
|
||||
end_index: Optional[int] = None,
|
||||
max_concurrency: int = 1,
|
||||
output_dir: str = "bench_results/longmemeval_reme_retrieve",
|
||||
reme_model_name: str = "qwen-flash",
|
||||
eval_model_name: str = "qwen3-max",
|
||||
algo_version: str = "v1",
|
||||
samples_per_type: int = -1,
|
||||
enable_thinking_params: bool = False,
|
||||
summary_results_dir: Optional[str] = None,
|
||||
):
|
||||
"""Main entry point for LongMemEval retrieve-only evaluation."""
|
||||
asyncio.run(
|
||||
main_async(
|
||||
data_path=data_path,
|
||||
top_k=top_k,
|
||||
start_index=start_index,
|
||||
end_index=end_index,
|
||||
max_concurrency=max_concurrency,
|
||||
output_dir=output_dir,
|
||||
reme_model_name=reme_model_name,
|
||||
eval_model_name=eval_model_name,
|
||||
algo_version=algo_version,
|
||||
samples_per_type=samples_per_type,
|
||||
enable_thinking_params=enable_thinking_params,
|
||||
summary_results_dir=summary_results_dir,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Evaluate ReMe on LongMemEval benchmark (Retrieve Phase Only)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--data_path",
|
||||
type=str,
|
||||
# default="/Users/zhouwk/PycharmProjects/MemAgent/dataset/longmemeval/longmemeval_s_cleaned.json",
|
||||
default="/Users/zhouwk/PycharmProjects/MemAgent/dataset/longmemeval/longmemeval_oracle.json",
|
||||
help="Path to LongMemEval JSON file",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--top_k",
|
||||
type=int,
|
||||
default=10,
|
||||
help="Number of memories to retrieve (default: 10)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--start_index",
|
||||
type=int,
|
||||
default=0,
|
||||
help="Start index for processing questions (default: 0)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--end_index",
|
||||
type=int,
|
||||
default=None,
|
||||
help="End index for processing questions (default: None, process all)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--max_concurrency",
|
||||
type=int,
|
||||
default=8,
|
||||
help="Maximum concurrent question processing (default: 1)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output_dir",
|
||||
type=str,
|
||||
default="bench_results/longmemeval_reme_retrieve_gpt4",
|
||||
help="Output directory for results",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--reme_model_name",
|
||||
type=str,
|
||||
default="gpt-4o-mini-2024-07-18",
|
||||
help="Model name for ReMe operations (default: gpt-4o-mini-2024-07-18)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--eval_model_name",
|
||||
type=str,
|
||||
default="gpt-4o-mini-2024-07-18",
|
||||
help="Model name for evaluation/judgment (default: gpt-4o-mini-2024-07-18)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--algo_version",
|
||||
type=str,
|
||||
default="longmemeval",
|
||||
help="Algorithm version for retrieval (default: longmemeval)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--samples_per_type",
|
||||
type=int,
|
||||
default=4,
|
||||
help="Number of samples per question type, -1 for all (default: 4)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--enable_thinking_params",
|
||||
action="store_true",
|
||||
default=False,
|
||||
help="Enable thinking parameters for retrieval (default: False)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--summary_results_dir",
|
||||
type=str,
|
||||
default="/Users/zhouwk/PycharmProjects/ReMe/benchmark/longmemeval/bench_results",
|
||||
help="Optional: path to previous summary results directory (for reference)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no_cache",
|
||||
action="store_true",
|
||||
default=False,
|
||||
help="Ignore cached results and re-run all questions (default: False)",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
print(f"args={args}!")
|
||||
|
||||
main(
|
||||
data_path=args.data_path,
|
||||
top_k=args.top_k,
|
||||
start_index=args.start_index,
|
||||
end_index=args.end_index,
|
||||
max_concurrency=args.max_concurrency,
|
||||
output_dir=args.output_dir,
|
||||
reme_model_name=args.reme_model_name,
|
||||
eval_model_name=args.eval_model_name,
|
||||
algo_version=args.algo_version,
|
||||
samples_per_type=args.samples_per_type,
|
||||
enable_thinking_params=args.enable_thinking_params,
|
||||
summary_results_dir=args.summary_results_dir,
|
||||
)
|
||||
548
benchmark/longmemeval/eval_reme.yaml
Normal file
548
benchmark/longmemeval/eval_reme.yaml
Normal file
|
|
@ -0,0 +1,548 @@
|
|||
TEMPLATE_MEMOS: |
|
||||
Memories for user {user_id}:
|
||||
{memories}
|
||||
|
||||
PROMPT_MEMZERO_JSON: |
|
||||
# CONTEXT:
|
||||
{context}
|
||||
|
||||
# CONTEXT PRIORITY:
|
||||
When the context contains information from multiple sources, follow this strict priority order:
|
||||
1. **Historical Dialogue** (highest priority) - Direct conversation content
|
||||
2. **Extracted Memories** (medium priority) - Summarized memory points
|
||||
3. **User Profile** (lowest priority) - General user information
|
||||
|
||||
# Question:
|
||||
{question}
|
||||
|
||||
# OUTPUT FORMAT:
|
||||
Do not hallucinate; strictly answer the user's question based on the content of the CONTEXT.
|
||||
Please provide your response in the following JSON format:
|
||||
|
||||
```json
|
||||
{{
|
||||
"reasoning": "reasoning content",
|
||||
"answer": "Provide a detailed answer"
|
||||
}}
|
||||
```
|
||||
|
||||
PROMPT_MEMZERO_JSON2: |
|
||||
# CONTEXT:
|
||||
{context}
|
||||
|
||||
# CONTEXT PRIORITY:
|
||||
When the context contains information from multiple sources, follow this strict priority order:
|
||||
1. **Historical Dialogue** (highest priority) - Direct conversation content
|
||||
2. **Extracted Memories** (medium priority) - Summarized memory points
|
||||
3. **User Profile** (lowest priority) - General user information
|
||||
|
||||
# Question:
|
||||
{question}
|
||||
|
||||
# OUTPUT FORMAT:
|
||||
Do not hallucinate; strictly answer the user's question based on the content of the CONTEXT.
|
||||
Please provide your response in the following JSON format:
|
||||
|
||||
```json
|
||||
{{
|
||||
"reasoning": "reasoning content",
|
||||
"answer": "Provide a detailed answer"
|
||||
}}
|
||||
```
|
||||
|
||||
PROMPT_MEMZERO: |
|
||||
You are an intelligent memory assistant tasked with retrieving accurate information from conversation memories.
|
||||
|
||||
# CONTEXT:
|
||||
You have access to memories from two speakers in a conversation. These memories contain
|
||||
timestamped information that may be relevant to answering the question.
|
||||
|
||||
# INSTRUCTIONS:
|
||||
1. Carefully analyze all provided memories from both speakers
|
||||
2. Pay special attention to the timestamps to determine the answer
|
||||
3. If the question asks about a specific event or fact, look for direct evidence in the memories
|
||||
4. If the memories contain contradictory information, prioritize the most recent memory
|
||||
5. If there is a question about time references (like "last year", "two months ago", etc.),
|
||||
calculate the actual date based on the memory timestamp. For example, if a memory from
|
||||
4 May 2022 mentions "went to India last year," then the trip occurred in 2021.
|
||||
6. Always convert relative time references to specific dates, months, or years. For example,
|
||||
convert "last year" to "2022" or "two months ago" to "March 2023" based on the memory
|
||||
timestamp. Ignore the reference while answering the question.
|
||||
7. Focus only on the content of the memories from both speakers. Do not confuse character
|
||||
names mentioned in memories with the actual users who created those memories.
|
||||
8. The answer should be less than 5-6 words.
|
||||
|
||||
# APPROACH (Think step by step):
|
||||
1. First, examine all memories that contain information related to the question
|
||||
2. Examine the timestamps and content of these memories carefully
|
||||
3. Look for explicit mentions of dates, times, locations, or events that answer the question
|
||||
4. If the answer requires calculation (e.g., converting relative time references), show your work
|
||||
5. Formulate a precise, concise answer based solely on the evidence in the memories
|
||||
6. Double-check that your answer directly addresses the question asked
|
||||
7. Ensure your final answer is specific and avoids vague time references
|
||||
|
||||
{context}
|
||||
|
||||
Question: {question}
|
||||
|
||||
Answer:
|
||||
|
||||
PROMPT_ZEP: |
|
||||
You are an intelligent memory assistant tasked with retrieving accurate information from conversation memories.
|
||||
|
||||
# CONTEXT:
|
||||
You have access to memories from a conversation. These memories contain
|
||||
timestamped information that may be relevant to answering the question.
|
||||
|
||||
# INSTRUCTIONS:
|
||||
1. Carefully analyze all provided memories
|
||||
2. Pay special attention to the timestamps to determine the answer
|
||||
3. If the question asks about a specific event or fact, look for direct evidence in the memories
|
||||
4. If the memories contain contradictory information, prioritize the most recent memory
|
||||
5. If there is a question about time references (like "last year", "two months ago", etc.),
|
||||
calculate the actual date based on the memory timestamp. For example, if a memory from
|
||||
4 May 2022 mentions "went to India last year," then the trip occurred in 2021.
|
||||
6. Always convert relative time references to specific dates, months, or years. For example,
|
||||
convert "last year" to "2022" or "two months ago" to "March 2023" based on the memory
|
||||
timestamp. Ignore the reference while answering the question.
|
||||
7. Focus only on the content of the memories. Do not confuse character
|
||||
names mentioned in memories with the actual users who created those memories.
|
||||
8. The answer should be less than 5-6 words.
|
||||
|
||||
# APPROACH (Think step by step):
|
||||
1. First, examine all memories that contain information related to the question
|
||||
2. Examine the timestamps and content of these memories carefully
|
||||
3. Look for explicit mentions of dates, times, locations, or events that answer the question
|
||||
4. If the answer requires calculation (e.g., converting relative time references), show your work
|
||||
5. Formulate a precise, concise answer based solely on the evidence in the memories
|
||||
6. Double-check that your answer directly addresses the question asked
|
||||
7. Ensure your final answer is specific and avoids vague time references
|
||||
|
||||
Context:
|
||||
|
||||
{context}
|
||||
|
||||
Question: {question}
|
||||
Answer:
|
||||
|
||||
PROMPT_MEMOS: |
|
||||
You are a knowledgeable and helpful AI assistant.
|
||||
|
||||
# CONTEXT:
|
||||
You have access to memories from two speakers in a conversation. These memories contain
|
||||
timestamped information that may be relevant to answering the question.
|
||||
|
||||
# INSTRUCTIONS:
|
||||
1. Carefully analyze all provided memories. Synthesize information across different entries if needed to form a complete answer.
|
||||
2. Pay close attention to the timestamps to determine the answer. If memories contain contradictory information, the **most recent memory** is the source of truth.
|
||||
3. If the question asks about a specific event or fact, look for direct evidence in the memories.
|
||||
4. Your answer must be grounded in the memories. However, you may use general world knowledge to interpret or complete information found within a memory (e.g., identifying a landmark mentioned by description).
|
||||
5. If the question involves time references (like "last year", "two months ago", etc.), you **must** calculate the actual date based on the memory's timestamp. For example, if a memory from 4 May 2022 mentions "went to India last year," then the trip occurred in 2021.
|
||||
6. Always convert relative time references to specific dates, months, or years in your final answer.
|
||||
7. Do not confuse character names mentioned in memories with the actual users who created them.
|
||||
8. The answer must be brief (under 5-6 words) and direct, with no extra description.
|
||||
|
||||
# APPROACH (Think step by step):
|
||||
1. First, examine all memories that contain information related to the question.
|
||||
2. Synthesize findings from multiple memories if a single entry is insufficient.
|
||||
3. Examine timestamps and content carefully, looking for explicit dates, times, locations, or events.
|
||||
4. If the answer requires calculation (e.g., converting relative time references), perform the calculation.
|
||||
5. Formulate a precise, concise answer based on the evidence from the memories (and allowed world knowledge).
|
||||
6. Double-check that your answer directly addresses the question asked and adheres to all instructions.
|
||||
7. Ensure your final answer is specific and avoids vague time references.
|
||||
|
||||
{context}
|
||||
|
||||
Question: {question}
|
||||
|
||||
Answer:
|
||||
|
||||
PROMPT_MEMOBASE: |
|
||||
You are an intelligent memory assistant tasked with retrieving accurate information from conversation memories.
|
||||
|
||||
# CONTEXT:
|
||||
You have access to memories from two speakers in a conversation. These memories contain
|
||||
timestamped information that may be relevant to answering the question.
|
||||
|
||||
# INSTRUCTIONS:
|
||||
1. Carefully analyze all provided memories from both speakers
|
||||
2. Pay special attention to the timestamps to determine the answer
|
||||
3. If the question asks about a specific event or fact, look for direct evidence in the memories
|
||||
4. If the memories contain contradictory information, prioritize the most recent memory
|
||||
5. If there is a question about time references (like "last year", "two months ago", etc.), calculate the actual date based on the memory timestamp. For example, if a memory from 4 May 2022 mentions "went to India last year," then the trip occurred in 2021.
|
||||
6. Always convert relative time references to specific dates, months, or years. For example, convert "last year" to "2022" or "two months ago" to "March 2023" based on the memory timestamp. Ignore the reference while answering the question.
|
||||
7. Focus only on the content of the memories from both speakers. Do not confuse character names mentioned in memories with the actual users who created those memories.
|
||||
8. The answer should be less than 5-6 words.
|
||||
|
||||
# APPROACH (Think step by step):
|
||||
1. First, examine all memories that contain information related to the question
|
||||
2. Examine the timestamps and content of these memories carefully
|
||||
3. Look for explicit mentions of dates, times, locations, or events that answer the question
|
||||
4. If the answer requires calculation (e.g., converting relative time references), show your work
|
||||
5. Formulate a precise, concise answer based solely on the evidence in the memories
|
||||
6. Double-check that your answer directly addresses the question asked
|
||||
7. Ensure your final answer is specific and avoids vague time references
|
||||
|
||||
{context}
|
||||
|
||||
Question: {question}
|
||||
|
||||
Answer:
|
||||
|
||||
|
||||
EVALUATION_PROMPT_FOR_MEMORY_INTEGRITY: |
|
||||
You are a strict **"Memory Integrity" evaluator**.
|
||||
Your core task is to assess whether an AI memory system has **missed any key memory points** after processing a conversation. This evaluation measures the system’s **memory integrity**, i.e., its ability to resist **amnesia** or **omission**.
|
||||
|
||||
# Evaluation Context & Data:
|
||||
|
||||
1. **Extracted Memories:**
|
||||
These are all the memory items actually extracted by the memory system.
|
||||
{memories}
|
||||
|
||||
2. **Expected Memory Point:**
|
||||
The key memory point that *should* have been extracted.
|
||||
{expected_memory_point}
|
||||
|
||||
# Evaluation Instructions:
|
||||
|
||||
1. For each **Expected Memory Point**, search within the **Extracted Memories** list for corresponding or related information. Ignore unrelated items.
|
||||
2. Based on the following scoring rubric, rate how well the memory system captured the **Expected Memory Point** and provide a detailed explanation.
|
||||
|
||||
# Scoring Rubric:
|
||||
|
||||
* **2:** Fully covered or implied.
|
||||
One or more items in “Extracted Memories” fully cover or logically imply all information in the “Expected Memory Point.”
|
||||
|
||||
* **1:** Partially covered or mentioned.
|
||||
Some information in “Extracted Memories” mentions part of the “Expected Memory Point,” but key information is missing, inaccurate, or slightly incorrect.
|
||||
|
||||
* **0:** Not mentioned or incorrect.
|
||||
“Extracted Memories” contains no mention of the “Expected Memory Point,” or the corresponding information is entirely wrong.
|
||||
|
||||
# Scoring Notes:
|
||||
|
||||
* For **compound Expected Memory Points** (with multiple elements such as person/event/time/location/preference, etc.):
|
||||
|
||||
* All elements correct → **2 points**
|
||||
* Some elements correct / uncertain → **1 point**
|
||||
* Key elements missing or wrong → **0 points**
|
||||
|
||||
* Semantic matching is acceptable; exact wording is **not** required.
|
||||
|
||||
* If “Extracted Memories” contains **conflicting information**, assign the **best possible coverage score** and mention the conflict in your reasoning.
|
||||
|
||||
* Extra or stylistically different memories do **not** reduce the score; only the coverage of the **Expected Memory Point** matters.
|
||||
|
||||
* For uncertain wording (“might,” “probably,” “tends to,” etc.):
|
||||
|
||||
* If the Expected Memory Point is a definite statement, usually assign **1 point**.
|
||||
|
||||
* If critical fields (e.g., time, entity name, relationship) are partly wrong but others match → **1 point**.
|
||||
|
||||
* If all key fields are wrong or missing → **0 points**.
|
||||
|
||||
# Output Format:
|
||||
|
||||
Please output your result in the following JSON format:
|
||||
|
||||
```json
|
||||
{{
|
||||
"reasoning": "Provide a concise justification for the score",
|
||||
"score": "2|1|0"
|
||||
}}
|
||||
```
|
||||
|
||||
EVALUATION_PROMPT_FOR_MEMORY_ACCURACY: |
|
||||
You are a **Dialogue Memory Accuracy Evaluator.** Your task is to evaluate the **accuracy** of a memory extracted by an AI memory system, based on three given inputs: the dialogue content, the *target (gold)* memory points (the correct annotated memories), and the *candidate* memory to be evaluated. The goal is to output a **structured evaluation result**.
|
||||
|
||||
# Input Content
|
||||
|
||||
* **Dialogue:**
|
||||
{dialogue}
|
||||
|
||||
* **Golden Memories (Target Memory Points):**
|
||||
The correct memory points pre-annotated for this dialogue in the evaluation dataset.
|
||||
{golden_memories}
|
||||
|
||||
* **Candidate Memory:**
|
||||
The memory extracted by the system to be evaluated.
|
||||
{candidate_memory}
|
||||
|
||||
# Evaluation Principles and Definitions
|
||||
|
||||
### 1) Support / Entailment
|
||||
|
||||
* An **information point** (atomic fact) in the candidate memory is considered *supported* if it can be directly stated or semantically entailed (via synonym, paraphrase, or equivalent expression) by the *Dialogue* or *Golden Memories*.
|
||||
* Only the given dialogue and golden memories can be used for judgment — **no external knowledge** or assumptions are allowed.
|
||||
Any information not appearing in or inferable from these two sources is considered *unsupported*.
|
||||
* Pay careful attention to **negation**, **quantities**, **time**, and **subjects**.
|
||||
If the candidate statement contradicts the dialogue or golden memories, it is considered a **conflict**.
|
||||
|
||||
### 2) Memory Accuracy Score (integer: 0 / 1 / 2)
|
||||
|
||||
* **2 points:** Every information point in the candidate memory is supported by the dialogue or golden memories, with **no contradictions or hallucinations**.
|
||||
* **1 point:** The candidate memory is *partially correct* (at least one supported information point) but also includes *unsupported* or *contradictory* content.
|
||||
* **0 points:** The candidate memory is **entirely unsupported or contradictory** to the sources (i.e., a “hallucinated memory”).
|
||||
|
||||
> Note:
|
||||
>
|
||||
> * If a candidate memory contains multiple information points, **any unsupported or contradictory element** prevents a full score (2).
|
||||
> * If both supported and unsupported/conflicting content appear, assign a score of **1**.
|
||||
|
||||
### 3) Inclusion in Golden Memories (Boolean field-level judgment)
|
||||
|
||||
**Definition:**
|
||||
|
||||
* **Atomic information point:** the smallest factual unit in the candidate memory (e.g., *name = Li Si*, *age = 25*, *location = Beijing*, *preference = coffee*, *budget ≤ 2000*, *meeting_time = Wednesday 10:00*, *tool = Zoom*, etc.).
|
||||
* **Field / Slot:** the semantic dimension of an information point (e.g., *name*, *age*, *residence*, *food preference*, *budget*, *meeting time*, *meeting tool*, etc.).
|
||||
|
||||
**Judgment Rules (independent of correctness):**
|
||||
|
||||
* **true:**
|
||||
Every atomic information point in the candidate memory has a corresponding **field** in the golden memories (allowing for synonyms, paraphrases, or equivalent expressions; ignore value, polarity, or quantity differences).
|
||||
|
||||
* Note: A single field in the gold list may match multiple candidate points (e.g., multiple “drink preference” facts can be covered by one “drink preference” field in gold).
|
||||
* **false:**
|
||||
If **any** atomic information point’s field in the candidate memory cannot be found in the golden memories, mark as *false*.
|
||||
|
||||
**Important Notes:**
|
||||
|
||||
* Field matching is restricted to fields that are **explicitly present or semantically recognizable** in the golden memories — no external knowledge may be used to expand the field set.
|
||||
* Differences in **values** (e.g., “Zhang San” vs. “Li Si”), **polarity** (like/dislike), or **exact number/time** do **not** affect this Boolean judgment.
|
||||
|
||||
# Evaluation Procedure
|
||||
|
||||
For each candidate memory:
|
||||
|
||||
1. **Decompose** it into atomic information points (e.g., name, number, location, preference).
|
||||
2. For each information point, **search** the dialogue and golden memories for supporting or contradictory evidence.
|
||||
3. Assign the **accuracy_score** (0 / 1 / 2) according to the rules above.
|
||||
4. Determine **is_included_in_golden_memories (true/false)**:
|
||||
|
||||
* Identify each information point’s field;
|
||||
* If *all* fields exist in the golden memories, mark as *true*; otherwise, *false*.
|
||||
5. Provide a **concise Chinese explanation** in `"reason"`, citing key evidence (short excerpts allowed), and clearly state any unsupported or contradictory parts if applicable.
|
||||
|
||||
# Output Format (strictly required)
|
||||
|
||||
Output **only one JSON object**, with the following three fields:
|
||||
|
||||
* `"accuracy_score"`: `"0"` or `"1"` or `"2"`
|
||||
* `"is_included_in_golden_memories"`: `"true"` or `"false"`
|
||||
* `"reason"`: `"brief explanation in Chinese"`
|
||||
|
||||
Do **not** include any other text, explanation, or fields.
|
||||
Do **not** include the candidate memory text inside the JSON.
|
||||
|
||||
Please output **only** the following JSON (in a code block):
|
||||
|
||||
```json
|
||||
{{
|
||||
"accuracy_score": "2 | 1 | 0",
|
||||
"is_included_in_golden_memories": "true | false",
|
||||
"reason": "Brief explanation in Chinese"
|
||||
}}
|
||||
```
|
||||
|
||||
EVALUATION_PROMPT_FOR_UPDATE_MEMORY: |
|
||||
Your task is to **evaluate the update accuracy** of an AI memory system.
|
||||
Based on the information provided below, determine whether the system-generated **“Generated Memories”** correctly **includes** the **Target Memory for Update**.
|
||||
|
||||
# Background Information
|
||||
|
||||
The following information is provided for evaluation:
|
||||
|
||||
1. **Generated Memories:**
|
||||
This is the list of memory points generated by the system after the current dialogue.
|
||||
{memories}
|
||||
|
||||
2. **Target Memory for Update:**
|
||||
This is the correct, updated version of the memory point that should have been produced — the one we focus on in this evaluation.
|
||||
{updated_memory}
|
||||
|
||||
3. **Original Memory Content:**
|
||||
This is the original version of the target memory before the update.
|
||||
{original_memory}
|
||||
|
||||
# Evaluation Criteria
|
||||
|
||||
Please make your judgment **strictly based on the content update of the “Target Memory for Update.”**
|
||||
Use the following categories:
|
||||
|
||||
### Correct Update
|
||||
|
||||
* **Generated Memories** **contains all information points** from the “Target Memory for Update,” accurately and completely reflecting the intended update.
|
||||
* **Key fields** (e.g., date, time, values, proper nouns, etc.) must match exactly.
|
||||
* The **original memory** is effectively replaced or marked as outdated.
|
||||
* Synonymous or slightly rephrased expressions are acceptable.
|
||||
|
||||
### Hallucinated Update
|
||||
|
||||
* **Factual error:** The **Generated Memories** includes a new memory related to the “Target Memory for Update,” but its content contains factual mistakes or contradictions compared to the correct update.
|
||||
|
||||
### Omitted Update
|
||||
|
||||
* **Completely omitted:** The **Generated Memories** contains no new memory related to the “Target Memory for Update.”
|
||||
* **Partially omitted:** A related new memory was generated in **Generated Memories**, but it **misses key information** that should have been included.
|
||||
|
||||
### Other
|
||||
|
||||
Used for update failures that do **not clearly fall** into the above categories of “Hallucination” or “Omission.”
|
||||
|
||||
# Output Requirements
|
||||
|
||||
Please return your evaluation strictly in the following JSON format and provide a concise explanation.
|
||||
|
||||
```json
|
||||
{{
|
||||
"reason": "Briefly explain your reasoning here and why it fits this category.",
|
||||
"evaluation_result": "Correct | Hallucination | Omission | Other"
|
||||
}}
|
||||
```
|
||||
|
||||
EVALUATION_PROMPT_FOR_QUESTION: |
|
||||
You are an **evaluation expert for AI memory system question answering**.
|
||||
Based **only** on the provided **“Question”**, **“Reference Answer”**, and **“Key Memory Points”** (the essential facts needed to derive the reference answer), strictly evaluate the **accuracy** of the **“Memory System Response.”** Classify it as one of **“Correct”**, **“Hallucination”**, or **“Omission.”** Do **not** use any external knowledge or subjective inference. Finally, output your judgment **strictly** in the specified JSON format.
|
||||
|
||||
# Evaluation Criteria
|
||||
|
||||
## Answer Type Classification
|
||||
|
||||
### 1. Correct
|
||||
|
||||
* The “Memory System Response” accurately answers the “Question,” and its content is **semantically equivalent** to the “Reference Answer.”
|
||||
* It contains **no contradictions** with the “Key Memory Points” or “Reference Answer.”
|
||||
* It introduces **no unsupported details** beyond the “Key Memory Points” that could alter the conclusion.
|
||||
* Synonyms, paraphrasing, and reasonable summarization are acceptable.
|
||||
|
||||
### 2. Hallucination
|
||||
|
||||
* The “Memory System Response” includes information or facts that **contradict or are inconsistent** with the “Reference Answer” or the “Key Memory Points.”
|
||||
* When the “Reference Answer” is labeled as *unknown/uncertain*, yet the response provides a specific verifiable fact or conclusion.
|
||||
* Extra irrelevant information that does **not change** the conclusion is **not** considered hallucination by itself; however, if it **changes or misleads** the conclusion, or **contradicts** the “Key Memory Points,” it should be judged as a **Hallucination**.
|
||||
|
||||
### 3. Omission
|
||||
|
||||
* The response is **incomplete** compared to the “Reference Answer.”
|
||||
* It explicitly states “don’t know,” “can’t remember,” or “no related memory,” even though relevant information exists in the “Key Memory Points.”
|
||||
* For multi-element questions, **all elements must be correct and present**; omission of **any** element is considered an **Omission**.
|
||||
|
||||
## Priority Rules (Conflict Handling)
|
||||
|
||||
* If the response contains **both missing necessary information** and **fabricated/contradictory information**, classify it as **Hallucination**.
|
||||
* If there is **no fabrication/contradiction** but some necessary information is missing, classify it as **Omission**.
|
||||
* Only when the meaning is **fully equivalent** to the reference answer should it be classified as **Correct**.
|
||||
|
||||
## Detailed Guidelines and Tolerance
|
||||
|
||||
* Equivalent expressions of numbers, times, and units are acceptable, but the **numerical values themselves must not differ**.
|
||||
* For multi-element questions, **all elements must be complete and accurate**; missing any element counts as **Omission**.
|
||||
* If the reference answer is *“unknown / cannot be determined”* and the system provides a definite fact, that is a **Hallucination**.
|
||||
If the system also answers *“unknown”* (without guessing), it may be **Correct**.
|
||||
* The evaluation must rely **only** on the *Reference Answer*, *Key Memory Points*, and *System Response* — no external context, world knowledge, or speculative reasoning is allowed.
|
||||
|
||||
# Information for Evaluation
|
||||
|
||||
* **Question:**
|
||||
{question}
|
||||
|
||||
* **Reference Answer:**
|
||||
{reference_answer}
|
||||
|
||||
* **Key Memory Points:**
|
||||
{key_memory_points}
|
||||
|
||||
* **Memory System Response:**
|
||||
{response}
|
||||
|
||||
# Output Requirements
|
||||
|
||||
Please provide your evaluation result **strictly** in the JSON format below.
|
||||
Do **not** add any extra explanation or comments outside the JSON block.
|
||||
|
||||
```json
|
||||
{{
|
||||
"reasoning": "Provide a concise and traceable evaluation rationale: first compare the system’s response with the Key Memory Points (which were correctly used, which were missing, and whether there was any fabrication/contradiction), then assess its consistency with the Reference Answer, and finally state the classification basis.",
|
||||
"evaluation_result": "Correct | Hallucination | Omission"
|
||||
}}
|
||||
```
|
||||
|
||||
|
||||
EVALUATION_PROMPT_FOR_QUESTION2: |
|
||||
You are an **evaluation expert for AI memory system question answering**.
|
||||
|
||||
Based **only** on the provided **"Question"**, **"Reference Answer"**, and **"Key Memory Points"** (the essential facts needed to derive the reference answer), strictly evaluate the **accuracy** of the **"Memory System Response."** Classify it as one of **"Correct"**, **"Hallucination"**, or **"Omission."** Do **not** use any external knowledge or subjective inference. Finally, output your judgment **strictly** in the specified JSON format.
|
||||
|
||||
# Evaluation Criteria
|
||||
|
||||
## Answer Type Classification
|
||||
|
||||
### 1. Correct
|
||||
|
||||
* The "Memory System Response" accurately answers the "Question," and its content is **semantically equivalent** to the "Reference Answer."
|
||||
* It contains **no contradictions** with the "Key Memory Points" or "Reference Answer."
|
||||
* **Extra details not present in the Key Memory Points are allowed and should not be penalized**, as long as they:
|
||||
- Do not contradict the Key Memory Points or Reference Answer
|
||||
- Do not change or mislead the core conclusion
|
||||
- Are reasonable additional context that the memory system may have retained from the conversation
|
||||
* The memory system may have stored additional information beyond the Key Memory Points. Such extra information should be treated as **supplementary context** rather than hallucination, provided it does not conflict with the core answer.
|
||||
* Synonyms, paraphrasing, and reasonable summarization are acceptable.
|
||||
|
||||
### 2. Hallucination
|
||||
|
||||
* The "Memory System Response" includes information or facts that **contradict or are inconsistent** with the "Reference Answer" or the "Key Memory Points."
|
||||
* The response provides information that **directly contradicts** known facts from the Key Memory Points.
|
||||
* When the "Reference Answer" is labeled as *unknown/uncertain*, yet the response provides a specific verifiable fact or conclusion.
|
||||
* **Important:** Extra information that is NOT in Key Memory Points is **NOT automatically a hallucination**. Only classify as hallucination if the extra information:
|
||||
- Directly contradicts the Key Memory Points or Reference Answer
|
||||
- Changes or misleads the core conclusion in a way that makes the answer incorrect
|
||||
- Provides a definitive answer when the Reference Answer indicates uncertainty
|
||||
|
||||
### 3. Omission
|
||||
|
||||
* The response is **incomplete** compared to the "Reference Answer."
|
||||
* It explicitly states "don't know," "can't remember," or "no related memory," even though relevant information exists in the "Key Memory Points."
|
||||
* For multi-element questions, **all elements must be correct and present**; omission of **any** element is considered an **Omission**.
|
||||
|
||||
## Priority Rules (Conflict Handling)
|
||||
|
||||
* If the response contains **both missing necessary information** and **fabricated/contradictory information**, classify it as **Hallucination**.
|
||||
* If there is **no fabrication/contradiction** but some necessary information is missing, classify it as **Omission**.
|
||||
* If the core answer is correct and complete, classify as **Correct** even if there are extra details not in Key Memory Points (as long as they don't contradict or mislead).
|
||||
|
||||
## Detailed Guidelines and Tolerance
|
||||
|
||||
* Equivalent expressions of numbers, times, and units are acceptable, but the **numerical values themselves must not differ**.
|
||||
* For multi-element questions, **all elements must be complete and accurate**; missing any element counts as **Omission**.
|
||||
* If the reference answer is *"unknown / cannot be determined"* and the system provides a definite fact, that is a **Hallucination**.
|
||||
If the system also answers *"unknown"* (without guessing), it may be **Correct**.
|
||||
* **Focus on evaluating whether the core answer to the question is correct**, not whether the response is limited to only the Key Memory Points.
|
||||
* Extra contextual information (e.g., additional preferences, related details) should be viewed as enrichment, not as errors, unless they contradict or mislead.
|
||||
|
||||
# Information for Evaluation
|
||||
|
||||
* **Question:**
|
||||
{question}
|
||||
|
||||
* **Reference Answer:**
|
||||
{reference_answer}
|
||||
|
||||
* **Key Memory Points:**
|
||||
{key_memory_points}
|
||||
|
||||
* **Memory System Response:**
|
||||
{response}
|
||||
|
||||
# Output Requirements
|
||||
|
||||
Please provide your evaluation result **strictly** in the JSON format below.
|
||||
Do **not** add any extra explanation or comments outside the JSON block.
|
||||
|
||||
```json
|
||||
{{
|
||||
"reasoning": "Provide a concise and traceable evaluation rationale: first verify that the system's response correctly includes all required elements from the Reference Answer, then check if any information contradicts the Key Memory Points or Reference Answer. Extra details not in Key Memory Points should be noted but not penalized unless they contradict or mislead. Finally state the classification basis.",
|
||||
"evaluation_result": "Correct | Hallucination | Omission"
|
||||
}}
|
||||
```
|
||||
"""
|
||||
232
benchmark/longmemeval/eval_tools.py
Normal file
232
benchmark/longmemeval/eval_tools.py
Normal file
|
|
@ -0,0 +1,232 @@
|
|||
"""Evaluation tools for ReMe LongMemEval benchmark."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
from reme.reme import ReMe
|
||||
|
||||
|
||||
# Load prompts from YAML file
|
||||
_YAML_PATH = Path(__file__).parent / "eval_reme.yaml"
|
||||
with open(_YAML_PATH, "r", encoding="utf-8") as f:
|
||||
_PROMPTS = yaml.safe_load(f)
|
||||
|
||||
|
||||
async def evaluation_for_memory_integrity(
|
||||
reme: ReMe,
|
||||
extract_memories: str,
|
||||
target_memory: str,
|
||||
model_name: str = "qwen3-max",
|
||||
) -> dict:
|
||||
"""
|
||||
Memory Integrity Evaluation
|
||||
|
||||
Args:
|
||||
reme: ReMe instance
|
||||
extract_memories: A formatted string concatenating all memory points extracted by the memory system.
|
||||
target_memory: The target key memory point.
|
||||
model_name: Model name for evaluation
|
||||
|
||||
Returns:
|
||||
dict with 'reasoning' and 'score' fields
|
||||
"""
|
||||
prompt = _PROMPTS["EVALUATION_PROMPT_FOR_MEMORY_INTEGRITY"].format(
|
||||
memories=extract_memories,
|
||||
expected_memory_point=target_memory,
|
||||
)
|
||||
|
||||
result = await reme.llm.simple_request_for_json(
|
||||
prompt=prompt,
|
||||
model_name=model_name,
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
async def evaluation_for_memory_accuracy(
|
||||
reme: ReMe,
|
||||
dialogue: str,
|
||||
golden_memories: str,
|
||||
candidate_memory: str,
|
||||
model_name: str = "qwen3-max",
|
||||
) -> dict:
|
||||
"""
|
||||
Memory Accuracy Evaluation
|
||||
|
||||
Args:
|
||||
reme: ReMe instance
|
||||
dialogue: The complete human-machine dialogue record.
|
||||
golden_memories: The core memory points for this dialogue segment in the evaluation set .
|
||||
candidate_memory: A specific memory point extracted by the memory system being evaluated.
|
||||
model_name: Model name for evaluation
|
||||
|
||||
Returns:
|
||||
dict with 'accuracy_score', 'is_included_in_golden_memories', and 'reason' fields
|
||||
"""
|
||||
prompt = _PROMPTS["EVALUATION_PROMPT_FOR_MEMORY_ACCURACY"].format(
|
||||
dialogue=dialogue,
|
||||
golden_memories=golden_memories,
|
||||
candidate_memory=candidate_memory,
|
||||
)
|
||||
|
||||
result = await reme.llm.simple_request_for_json(
|
||||
prompt=prompt,
|
||||
model_name=model_name,
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
async def evaluation_for_update_memory(
|
||||
reme: ReMe,
|
||||
extract_memories: str,
|
||||
target_update_memory: str,
|
||||
original_memory: str,
|
||||
model_name: str = "qwen3-max",
|
||||
) -> dict:
|
||||
"""
|
||||
Memory Update Evaluation
|
||||
|
||||
Args:
|
||||
reme: ReMe instance
|
||||
extract_memories: A formatted string concatenating all memory points extracted by the memory system .
|
||||
target_update_memory: The target updated memory point.
|
||||
original_memory: A formatted string concatenating all original memory points corresponding.
|
||||
model_name: Model name for evaluation
|
||||
|
||||
Returns:
|
||||
dict with 'reason' and 'evaluation_result' fields
|
||||
"""
|
||||
prompt = _PROMPTS["EVALUATION_PROMPT_FOR_UPDATE_MEMORY"].format(
|
||||
memories=extract_memories,
|
||||
updated_memory=target_update_memory,
|
||||
original_memory=original_memory,
|
||||
)
|
||||
|
||||
result = await reme.llm.simple_request_for_json(
|
||||
prompt=prompt,
|
||||
model_name=model_name,
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
async def evaluation_for_question(
|
||||
reme: ReMe,
|
||||
question: str,
|
||||
reference_answer: str,
|
||||
key_memory_points: str,
|
||||
response: str,
|
||||
model_name: str = "qwen3-max",
|
||||
) -> dict:
|
||||
"""
|
||||
Question-Answering Evaluation
|
||||
|
||||
Args:
|
||||
reme: ReMe instance
|
||||
question: The question string to be evaluated.
|
||||
reference_answer: The reference (gold-standard) answer.
|
||||
key_memory_points: The memory points used to derive the reference answer.
|
||||
response: The answer produced by the memory system.
|
||||
model_name: Model name for evaluation
|
||||
|
||||
Returns:
|
||||
dict with 'reasoning' and 'evaluation_result' fields
|
||||
"""
|
||||
prompt = _PROMPTS["EVALUATION_PROMPT_FOR_QUESTION"].format(
|
||||
question=question,
|
||||
reference_answer=reference_answer,
|
||||
key_memory_points=key_memory_points,
|
||||
response=response,
|
||||
)
|
||||
|
||||
result = await reme.llm.simple_request_for_json(
|
||||
prompt=prompt,
|
||||
model_name=model_name,
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
async def evaluation_for_question2(
|
||||
reme: ReMe,
|
||||
question: str,
|
||||
reference_answer: str,
|
||||
key_memory_points: str,
|
||||
response: str,
|
||||
dialogue: str = "",
|
||||
model_name: str = "qwen3-max",
|
||||
) -> dict:
|
||||
"""
|
||||
Question-Answering Evaluation with Dialogue Context (Version 2)
|
||||
|
||||
Args:
|
||||
reme: ReMe instance
|
||||
question: The question string to be evaluated.
|
||||
reference_answer: The reference (gold-standard) answer.
|
||||
key_memory_points: The memory points used to derive the reference answer.
|
||||
response: The answer produced by the memory system.
|
||||
dialogue: The formatted dialogue history (role, content, time_created).
|
||||
model_name: Model name for evaluation
|
||||
|
||||
Returns:
|
||||
dict with 'reasoning' and 'evaluation_result' fields
|
||||
"""
|
||||
prompt = _PROMPTS["EVALUATION_PROMPT_FOR_QUESTION2"].format(
|
||||
question=question,
|
||||
reference_answer=reference_answer,
|
||||
key_memory_points=key_memory_points,
|
||||
response=response,
|
||||
dialogue=dialogue if dialogue else "",
|
||||
)
|
||||
|
||||
result = await reme.llm.simple_request_for_json(
|
||||
prompt=prompt,
|
||||
model_name=model_name,
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
async def answer_question_with_memories(
|
||||
reme: ReMe,
|
||||
question: str,
|
||||
memories: str,
|
||||
user_id: str = None,
|
||||
model_name: str = "qwen3-max",
|
||||
) -> dict:
|
||||
"""
|
||||
Answer a question using retrieved memories with PROMPT_MEMZERO_JSON template.
|
||||
|
||||
Args:
|
||||
reme: ReMe instance
|
||||
question: The question to answer
|
||||
memories: The retrieved memories (formatted as context)
|
||||
user_id: Optional user ID for context formatting
|
||||
model_name: Model name for LLM request
|
||||
|
||||
Returns:
|
||||
dict with 'reasoning' and 'answer' fields
|
||||
"""
|
||||
# Format context with memories
|
||||
if user_id:
|
||||
context = _PROMPTS["TEMPLATE_MEMOS"].format(
|
||||
user_id=user_id,
|
||||
memories=memories,
|
||||
)
|
||||
else:
|
||||
context = f"Memories:\n{memories}"
|
||||
|
||||
# Use PROMPT_MEMZERO_JSON template for structured JSON response
|
||||
prompt = _PROMPTS["PROMPT_MEMZERO_JSON"].format(
|
||||
context=context,
|
||||
question=question,
|
||||
)
|
||||
|
||||
result = await reme.llm.simple_request_for_json(
|
||||
prompt=prompt,
|
||||
model_name=model_name,
|
||||
)
|
||||
|
||||
return result
|
||||
97
benchmark/longmemeval/llms.py
Normal file
97
benchmark/longmemeval/llms.py
Normal file
|
|
@ -0,0 +1,97 @@
|
|||
"""LLM utilities for LongMemEval benchmark evaluation."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
|
||||
from tenacity import retry, stop_after_attempt, wait_random_exponential, before_sleep_log
|
||||
|
||||
from reme.core.schema import Message
|
||||
from reme.core.utils import load_env
|
||||
from reme.reme import ReMe
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
load_env()
|
||||
|
||||
WAIT_TIME_LOWER = 1
|
||||
WAIT_TIME_UPPER = 60
|
||||
RETRY_TIMES = 5
|
||||
|
||||
|
||||
@retry(
|
||||
wait=wait_random_exponential(min=WAIT_TIME_LOWER, max=WAIT_TIME_UPPER),
|
||||
stop=stop_after_attempt(3),
|
||||
reraise=True,
|
||||
before_sleep=before_sleep_log(logger, logging.WARNING),
|
||||
)
|
||||
async def llm_request(reme: ReMe, prompt: str, model_name: str = "qwen3-max", **kwargs) -> str:
|
||||
"""Make an LLM request using ReMe's LLM with optional model override.
|
||||
|
||||
Args:
|
||||
reme: ReMe instance
|
||||
prompt: The prompt to send to the LLM
|
||||
model_name: Optional model name to override the default model (default: "qwen3-max")
|
||||
**kwargs: Additional arguments to pass to the chat method
|
||||
|
||||
Returns:
|
||||
The assistant's response content
|
||||
"""
|
||||
assistant_message = await reme.llm.chat(
|
||||
messages=[
|
||||
Message(role="user", content=prompt),
|
||||
],
|
||||
model_name=model_name,
|
||||
**kwargs,
|
||||
)
|
||||
return assistant_message.content
|
||||
|
||||
|
||||
@retry(
|
||||
wait=wait_random_exponential(min=WAIT_TIME_LOWER, max=WAIT_TIME_UPPER),
|
||||
stop=stop_after_attempt(RETRY_TIMES),
|
||||
reraise=True,
|
||||
before_sleep=before_sleep_log(logger, logging.WARNING),
|
||||
)
|
||||
async def llm_request_for_json(reme: ReMe, prompt: str, model_name: str = "qwen-flash", **kwargs) -> dict:
|
||||
"""Make an LLM request expecting JSON response using ReMe's LLM.
|
||||
|
||||
Args:
|
||||
reme: ReMe instance
|
||||
prompt: The prompt to send to the LLM
|
||||
model_name: Optional model name to override the default model (default: "qwen-flash")
|
||||
**kwargs: Additional arguments to pass to the chat method
|
||||
|
||||
Returns:
|
||||
Parsed JSON object from the LLM response
|
||||
|
||||
Raises:
|
||||
ValueError: If no JSON block is found in the model output
|
||||
"""
|
||||
content = await llm_request(reme, prompt, model_name=model_name, **kwargs)
|
||||
|
||||
match = re.search(r"```json\s*(\{.*?\})\s*```", content, re.DOTALL)
|
||||
if not match:
|
||||
raise ValueError(f"No JSON block found in model output: {content}")
|
||||
|
||||
json_str = match.group(1).strip()
|
||||
return json.loads(json_str)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
async def test():
|
||||
"""Simple manual test for JSON LLM request."""
|
||||
reme = ReMe()
|
||||
await reme.start()
|
||||
try:
|
||||
r = await llm_request_for_json(
|
||||
reme,
|
||||
'hello? answer in ```json\n{"answer": "..."}```',
|
||||
)
|
||||
print(r)
|
||||
finally:
|
||||
await reme.close()
|
||||
|
||||
asyncio.run(test())
|
||||
Loading…
Add table
Reference in a new issue