mirror of
https://github.com/agentscope-ai/ReMe.git
synced 2026-10-17 04:37:00 +00:00
145 lines
No EOL
8.8 KiB
YAML
145 lines
No EOL
8.8 KiB
YAML
EVALUATION_PROMPT_FOR_QUESTION: |
|
||
You are an **evaluation expert for AI memory system question answering**.
|
||
Based **only** on the provided **“Question”**, **“Reference Answer”**, and **“Key Memory Points”** (the essential facts needed to derive the reference answer), strictly evaluate the **accuracy** of the **“Memory System Response.”** Classify it as one of **“Correct”**, **“Hallucination”**, or **“Omission.”** Do **not** use any external knowledge or subjective inference. Finally, output your judgment **strictly** in the specified JSON format.
|
||
|
||
# Evaluation Criteria
|
||
|
||
## Answer Type Classification
|
||
|
||
### 1. Correct
|
||
|
||
* The “Memory System Response” accurately answers the “Question,” and its content is **semantically equivalent** to the “Reference Answer.”
|
||
* It contains **no contradictions** with the “Key Memory Points” or “Reference Answer.”
|
||
* It introduces **no unsupported details** beyond the “Key Memory Points” that could alter the conclusion.
|
||
* Synonyms, paraphrasing, and reasonable summarization are acceptable.
|
||
|
||
### 2. Hallucination
|
||
|
||
* The “Memory System Response” includes information or facts that **contradict or are inconsistent** with the “Reference Answer” or the “Key Memory Points.”
|
||
* When the “Reference Answer” is labeled as *unknown/uncertain*, yet the response provides a specific verifiable fact or conclusion.
|
||
* Extra irrelevant information that does **not change** the conclusion is **not** considered hallucination by itself; however, if it **changes or misleads** the conclusion, or **contradicts** the “Key Memory Points,” it should be judged as a **Hallucination**.
|
||
|
||
### 3. Omission
|
||
|
||
* The response is **incomplete** compared to the “Reference Answer.”
|
||
* It explicitly states “don’t know,” “can’t remember,” or “no related memory,” even though relevant information exists in the “Key Memory Points.”
|
||
* For multi-element questions, **all elements must be correct and present**; omission of **any** element is considered an **Omission**.
|
||
|
||
## Priority Rules (Conflict Handling)
|
||
|
||
* If the response contains **both missing necessary information** and **fabricated/contradictory information**, classify it as **Hallucination**.
|
||
* If there is **no fabrication/contradiction** but some necessary information is missing, classify it as **Omission**.
|
||
* Only when the meaning is **fully equivalent** to the reference answer should it be classified as **Correct**.
|
||
|
||
## Detailed Guidelines and Tolerance
|
||
|
||
* Equivalent expressions of numbers, times, and units are acceptable, but the **numerical values themselves must not differ**.
|
||
* For multi-element questions, **all elements must be complete and accurate**; missing any element counts as **Omission**.
|
||
* If the reference answer is *“unknown / cannot be determined”* and the system provides a definite fact, that is a **Hallucination**.
|
||
If the system also answers *“unknown”* (without guessing), it may be **Correct**.
|
||
* The evaluation must rely **only** on the *Reference Answer*, *Key Memory Points*, and *System Response* — no external context, world knowledge, or speculative reasoning is allowed.
|
||
|
||
# Information for Evaluation
|
||
|
||
* **Question:**
|
||
{question}
|
||
|
||
* **Reference Answer:**
|
||
{reference_answer}
|
||
|
||
* **Key Memory Points:**
|
||
{key_memory_points}
|
||
|
||
* **Memory System Response:**
|
||
{response}
|
||
|
||
# Output Requirements
|
||
|
||
Please provide your evaluation result **strictly** in the JSON format below.
|
||
Do **not** add any extra explanation or comments outside the JSON block.
|
||
|
||
```json
|
||
{{
|
||
"reasoning": "Provide a concise and traceable evaluation rationale: first compare the system’s response with the Key Memory Points (which were correctly used, which were missing, and whether there was any fabrication/contradiction), then assess its consistency with the Reference Answer, and finally state the classification basis.",
|
||
"evaluation_result": "Correct | Hallucination | Omission"
|
||
}}
|
||
```
|
||
|
||
|
||
EVALUATION_PROMPT_FOR_QUESTION2: |
|
||
You are an **evaluation expert for AI memory system question answering**.
|
||
|
||
Based **only** on the provided **"Question"**, **"Reference Answer"**, and **"Key Memory Points"** (the essential facts needed to derive the reference answer), strictly evaluate the **accuracy** of the **"Memory System Response."** Classify it as one of **"Correct"**, **"Hallucination"**, or **"Omission."** Do **not** use any external knowledge or subjective inference. Finally, output your judgment **strictly** in the specified JSON format.
|
||
|
||
# Evaluation Criteria
|
||
|
||
## Answer Type Classification
|
||
|
||
### 1. Correct
|
||
|
||
* The "Memory System Response" accurately answers the "Question," and its content is **semantically equivalent** to the "Reference Answer."
|
||
* It contains **no contradictions** with the "Key Memory Points" or "Reference Answer."
|
||
* **Extra details not present in the Key Memory Points are allowed and should not be penalized**, as long as they:
|
||
- Do not contradict the Key Memory Points or Reference Answer
|
||
- Do not change or mislead the core conclusion
|
||
- Are reasonable additional context that the memory system may have retained from the conversation
|
||
* The memory system may have stored additional information beyond the Key Memory Points. Such extra information should be treated as **supplementary context** rather than hallucination, provided it does not conflict with the core answer.
|
||
* Synonyms, paraphrasing, and reasonable summarization are acceptable.
|
||
|
||
### 2. Hallucination
|
||
|
||
* The "Memory System Response" includes information or facts that **contradict or are inconsistent** with the "Reference Answer" or the "Key Memory Points."
|
||
* The response provides information that **directly contradicts** known facts from the Key Memory Points.
|
||
* When the "Reference Answer" is labeled as *unknown/uncertain*, yet the response provides a specific verifiable fact or conclusion.
|
||
* **Important:** Extra information that is NOT in Key Memory Points is **NOT automatically a hallucination**. Only classify as hallucination if the extra information:
|
||
- Directly contradicts the Key Memory Points or Reference Answer
|
||
- Changes or misleads the core conclusion in a way that makes the answer incorrect
|
||
- Provides a definitive answer when the Reference Answer indicates uncertainty
|
||
|
||
### 3. Omission
|
||
|
||
* The response is **incomplete** compared to the "Reference Answer."
|
||
* It explicitly states "don't know," "can't remember," or "no related memory," even though relevant information exists in the "Key Memory Points."
|
||
* For multi-element questions, **all elements must be correct and present**; omission of **any** element is considered an **Omission**.
|
||
|
||
## Priority Rules (Conflict Handling)
|
||
|
||
* If the response contains **both missing necessary information** and **fabricated/contradictory information**, classify it as **Hallucination**.
|
||
* If there is **no fabrication/contradiction** but some necessary information is missing, classify it as **Omission**.
|
||
* If the core answer is correct and complete, classify as **Correct** even if there are extra details not in Key Memory Points (as long as they don't contradict or mislead).
|
||
|
||
## Detailed Guidelines and Tolerance
|
||
|
||
* Equivalent expressions of numbers, times, and units are acceptable, but the **numerical values themselves must not differ**.
|
||
* For multi-element questions, **all elements must be complete and accurate**; missing any element counts as **Omission**.
|
||
* If the reference answer is *"unknown / cannot be determined"* and the system provides a definite fact, that is a **Hallucination**.
|
||
If the system also answers *"unknown"* (without guessing), it may be **Correct**.
|
||
* **Focus on evaluating whether the core answer to the question is correct**, not whether the response is limited to only the Key Memory Points.
|
||
* Extra contextual information (e.g., additional preferences, related details) should be viewed as enrichment, not as errors, unless they contradict or mislead.
|
||
|
||
# Information for Evaluation
|
||
|
||
* **Question:**
|
||
{question}
|
||
|
||
* **Reference Answer:**
|
||
{reference_answer}
|
||
|
||
* **Key Memory Points:**
|
||
{key_memory_points}
|
||
|
||
* **Memory System Response:**
|
||
{response}
|
||
|
||
# Output Requirements
|
||
|
||
Please provide your evaluation result **strictly** in the JSON format below.
|
||
Do **not** add any extra explanation or comments outside the JSON block.
|
||
|
||
```json
|
||
{{
|
||
"reasoning": "Provide a concise and traceable evaluation rationale: first verify that the system's response correctly includes all required elements from the Reference Answer, then check if any information contradicts the Key Memory Points or Reference Answer. Extra details not in Key Memory Points should be noted but not penalized unless they contradict or mislead. Finally state the classification basis.",
|
||
"evaluation_result": "Correct | Hallucination | Omission"
|
||
}}
|
||
```
|
||
""" |