TEMPLATE_MEMOS: | Memories for user {user_id}: {memories} PROMPT_MEMZERO_JSON: | # CONTEXT: {context} # CONTEXT PRIORITY: When the context contains information from multiple sources, follow this strict priority order: 1. **Historical Dialogue** (highest priority) - Direct conversation content 2. **Extracted Memories** (medium priority) - Summarized memory points 3. **User Profile** (lowest priority) - General user information # Question: {question} # OUTPUT FORMAT: Do not hallucinate; strictly answer the user's question based on the content of the CONTEXT. Please provide your response in the following JSON format: ```json {{ "reasoning": "reasoning content", "answer": "Provide a detailed answer" }} ``` PROMPT_MEMZERO_JSON2: | # CONTEXT: {context} # CONTEXT PRIORITY: When the context contains information from multiple sources, follow this strict priority order: 1. **Historical Dialogue** (highest priority) - Direct conversation content 2. **Extracted Memories** (medium priority) - Summarized memory points 3. **User Profile** (lowest priority) - General user information # Question: {question} # OUTPUT FORMAT: Do not hallucinate; strictly answer the user's question based on the content of the CONTEXT. Please provide your response in the following JSON format: ```json {{ "reasoning": "reasoning content", "answer": "Provide a detailed answer" }} ``` PROMPT_MEMZERO: | You are an intelligent memory assistant tasked with retrieving accurate information from conversation memories. # CONTEXT: You have access to memories from two speakers in a conversation. These memories contain timestamped information that may be relevant to answering the question. # INSTRUCTIONS: 1. Carefully analyze all provided memories from both speakers 2. Pay special attention to the timestamps to determine the answer 3. If the question asks about a specific event or fact, look for direct evidence in the memories 4. If the memories contain contradictory information, prioritize the most recent memory 5. If there is a question about time references (like "last year", "two months ago", etc.), calculate the actual date based on the memory timestamp. For example, if a memory from 4 May 2022 mentions "went to India last year," then the trip occurred in 2021. 6. Always convert relative time references to specific dates, months, or years. For example, convert "last year" to "2022" or "two months ago" to "March 2023" based on the memory timestamp. Ignore the reference while answering the question. 7. Focus only on the content of the memories from both speakers. Do not confuse character names mentioned in memories with the actual users who created those memories. 8. The answer should be less than 5-6 words. # APPROACH (Think step by step): 1. First, examine all memories that contain information related to the question 2. Examine the timestamps and content of these memories carefully 3. Look for explicit mentions of dates, times, locations, or events that answer the question 4. If the answer requires calculation (e.g., converting relative time references), show your work 5. Formulate a precise, concise answer based solely on the evidence in the memories 6. Double-check that your answer directly addresses the question asked 7. Ensure your final answer is specific and avoids vague time references {context} Question: {question} Answer: PROMPT_ZEP: | You are an intelligent memory assistant tasked with retrieving accurate information from conversation memories. # CONTEXT: You have access to memories from a conversation. These memories contain timestamped information that may be relevant to answering the question. # INSTRUCTIONS: 1. Carefully analyze all provided memories 2. Pay special attention to the timestamps to determine the answer 3. If the question asks about a specific event or fact, look for direct evidence in the memories 4. If the memories contain contradictory information, prioritize the most recent memory 5. If there is a question about time references (like "last year", "two months ago", etc.), calculate the actual date based on the memory timestamp. For example, if a memory from 4 May 2022 mentions "went to India last year," then the trip occurred in 2021. 6. Always convert relative time references to specific dates, months, or years. For example, convert "last year" to "2022" or "two months ago" to "March 2023" based on the memory timestamp. Ignore the reference while answering the question. 7. Focus only on the content of the memories. Do not confuse character names mentioned in memories with the actual users who created those memories. 8. The answer should be less than 5-6 words. # APPROACH (Think step by step): 1. First, examine all memories that contain information related to the question 2. Examine the timestamps and content of these memories carefully 3. Look for explicit mentions of dates, times, locations, or events that answer the question 4. If the answer requires calculation (e.g., converting relative time references), show your work 5. Formulate a precise, concise answer based solely on the evidence in the memories 6. Double-check that your answer directly addresses the question asked 7. Ensure your final answer is specific and avoids vague time references Context: {context} Question: {question} Answer: PROMPT_MEMOS: | You are a knowledgeable and helpful AI assistant. # CONTEXT: You have access to memories from two speakers in a conversation. These memories contain timestamped information that may be relevant to answering the question. # INSTRUCTIONS: 1. Carefully analyze all provided memories. Synthesize information across different entries if needed to form a complete answer. 2. Pay close attention to the timestamps to determine the answer. If memories contain contradictory information, the **most recent memory** is the source of truth. 3. If the question asks about a specific event or fact, look for direct evidence in the memories. 4. Your answer must be grounded in the memories. However, you may use general world knowledge to interpret or complete information found within a memory (e.g., identifying a landmark mentioned by description). 5. If the question involves time references (like "last year", "two months ago", etc.), you **must** calculate the actual date based on the memory's timestamp. For example, if a memory from 4 May 2022 mentions "went to India last year," then the trip occurred in 2021. 6. Always convert relative time references to specific dates, months, or years in your final answer. 7. Do not confuse character names mentioned in memories with the actual users who created them. 8. The answer must be brief (under 5-6 words) and direct, with no extra description. # APPROACH (Think step by step): 1. First, examine all memories that contain information related to the question. 2. Synthesize findings from multiple memories if a single entry is insufficient. 3. Examine timestamps and content carefully, looking for explicit dates, times, locations, or events. 4. If the answer requires calculation (e.g., converting relative time references), perform the calculation. 5. Formulate a precise, concise answer based on the evidence from the memories (and allowed world knowledge). 6. Double-check that your answer directly addresses the question asked and adheres to all instructions. 7. Ensure your final answer is specific and avoids vague time references. {context} Question: {question} Answer: PROMPT_MEMOBASE: | You are an intelligent memory assistant tasked with retrieving accurate information from conversation memories. # CONTEXT: You have access to memories from two speakers in a conversation. These memories contain timestamped information that may be relevant to answering the question. # INSTRUCTIONS: 1. Carefully analyze all provided memories from both speakers 2. Pay special attention to the timestamps to determine the answer 3. If the question asks about a specific event or fact, look for direct evidence in the memories 4. If the memories contain contradictory information, prioritize the most recent memory 5. If there is a question about time references (like "last year", "two months ago", etc.), calculate the actual date based on the memory timestamp. For example, if a memory from 4 May 2022 mentions "went to India last year," then the trip occurred in 2021. 6. Always convert relative time references to specific dates, months, or years. For example, convert "last year" to "2022" or "two months ago" to "March 2023" based on the memory timestamp. Ignore the reference while answering the question. 7. Focus only on the content of the memories from both speakers. Do not confuse character names mentioned in memories with the actual users who created those memories. 8. The answer should be less than 5-6 words. # APPROACH (Think step by step): 1. First, examine all memories that contain information related to the question 2. Examine the timestamps and content of these memories carefully 3. Look for explicit mentions of dates, times, locations, or events that answer the question 4. If the answer requires calculation (e.g., converting relative time references), show your work 5. Formulate a precise, concise answer based solely on the evidence in the memories 6. Double-check that your answer directly addresses the question asked 7. Ensure your final answer is specific and avoids vague time references {context} Question: {question} Answer: EVALUATION_PROMPT_FOR_MEMORY_INTEGRITY: | You are a strict **"Memory Integrity" evaluator**. Your core task is to assess whether an AI memory system has **missed any key memory points** after processing a conversation. This evaluation measures the system's **memory integrity**, i.e., its ability to resist **amnesia** or **omission**. # Evaluation Context & Data: 1. **Extracted Memories:** These are all the memory items actually extracted by the memory system. {memories} 2. **Expected Memory Point:** The key memory point that *should* have been extracted. {expected_memory_point} # Evaluation Instructions: 1. For each **Expected Memory Point**, search within the **Extracted Memories** list for corresponding or related information. Ignore unrelated items. 2. Based on the following scoring rubric, rate how well the memory system captured the **Expected Memory Point** and provide a detailed explanation. # Scoring Rubric: * **2:** Fully covered or implied. One or more items in "Extracted Memories" fully cover or logically imply all information in the "Expected Memory Point." * **1:** Partially covered or mentioned. Some information in "Extracted Memories" mentions part of the "Expected Memory Point," but key information is missing, inaccurate, or slightly incorrect. * **0:** Not mentioned or incorrect. "Extracted Memories" contains no mention of the "Expected Memory Point," or the corresponding information is entirely wrong. # Scoring Notes: * For **compound Expected Memory Points** (with multiple elements such as person/event/time/location/preference, etc.): * All elements correct → **2 points** * Some elements correct / uncertain → **1 point** * Key elements missing or wrong → **0 points** * Semantic matching is acceptable; exact wording is **not** required. * If "Extracted Memories" contains **conflicting information**, assign the **best possible coverage score** and mention the conflict in your reasoning. * Extra or stylistically different memories do **not** reduce the score; only the coverage of the **Expected Memory Point** matters. * For uncertain wording ("might," "probably," "tends to," etc.): * If the Expected Memory Point is a definite statement, usually assign **1 point**. * If critical fields (e.g., time, entity name, relationship) are partly wrong but others match → **1 point**. * If all key fields are wrong or missing → **0 points**. # Output Format: Please output your result in the following JSON format: ```json {{ "reasoning": "Provide a concise justification for the score", "score": "2|1|0" }} ``` EVALUATION_PROMPT_FOR_MEMORY_ACCURACY: | You are a **Dialogue Memory Accuracy Evaluator.** Your task is to evaluate the **accuracy** of a memory extracted by an AI memory system, based on three given inputs: the dialogue content, the *target (gold)* memory points (the correct annotated memories), and the *candidate* memory to be evaluated. The goal is to output a **structured evaluation result**. # Input Content * **Dialogue:** {dialogue} * **Golden Memories (Target Memory Points):** The correct memory points pre-annotated for this dialogue in the evaluation dataset. {golden_memories} * **Candidate Memory:** The memory extracted by the system to be evaluated. {candidate_memory} # Evaluation Principles and Definitions ### 1) Support / Entailment * An **information point** (atomic fact) in the candidate memory is considered *supported* if it can be directly stated or semantically entailed (via synonym, paraphrase, or equivalent expression) by the *Dialogue* or *Golden Memories*. * Only the given dialogue and golden memories can be used for judgment — **no external knowledge** or assumptions are allowed. Any information not appearing in or inferable from these two sources is considered *unsupported*. * Pay careful attention to **negation**, **quantities**, **time**, and **subjects**. If the candidate statement contradicts the dialogue or golden memories, it is considered a **conflict**. ### 2) Memory Accuracy Score (integer: 0 / 1 / 2) * **2 points:** Every information point in the candidate memory is supported by the dialogue or golden memories, with **no contradictions or hallucinations**. * **1 point:** The candidate memory is *partially correct* (at least one supported information point) but also includes *unsupported* or *contradictory* content. * **0 points:** The candidate memory is **entirely unsupported or contradictory** to the sources (i.e., a "hallucinated memory"). > Note: > * If a candidate memory contains multiple information points, **any unsupported or contradictory element** prevents a full score (2). > * If both supported and unsupported/conflicting content appear, assign a score of **1**. ### 3) Inclusion in Golden Memories (Boolean field-level judgment) **Definition:** * **Atomic information point:** the smallest factual unit in the candidate memory (e.g., *name = Li Si*, *age = 25*, *location = Beijing*, *preference = coffee*, *budget ≤ 2000*, *meeting_time = Wednesday 10:00*, *tool = Zoom*, etc.). * **Field / Slot:** the semantic dimension of an information point (e.g., *name*, *age*, *residence*, *food preference*, *budget*, *meeting time*, *meeting tool*, etc.). **Judgment Rules (independent of correctness):** * **true:** Every atomic information point in the candidate memory has a corresponding **field** in the golden memories (allowing for synonyms, paraphrases, or equivalent expressions; ignore value, polarity, or quantity differences). * Note: A single field in the gold list may match multiple candidate points (e.g., multiple "drink preference" facts can be covered by one "drink preference" field in gold). * **false:** If **any** atomic information point's field in the candidate memory cannot be found in the golden memories, mark as *false*. **Important Notes:** * Field matching is restricted to fields that are **explicitly present or semantically recognizable** in the golden memories — no external knowledge may be used to expand the field set. * Differences in **values** (e.g., "Zhang San" vs. "Li Si"), **polarity** (like/dislike), or **exact number/time** do **not** affect this Boolean judgment. # Evaluation Procedure For each candidate memory: 1. **Decompose** it into atomic information points (e.g., name, number, location, preference). 2. For each information point, **search** the dialogue and golden memories for supporting or contradictory evidence. 3. Assign the **accuracy_score** (0 / 1 / 2) according to the rules above. 4. Determine **is_included_in_golden_memories (true/false)**: * Identify each information point's field; * If *all* fields exist in the golden memories, mark as *true*; otherwise, *false*. 5. Provide a **concise Chinese explanation** in `"reason"`, citing key evidence (short excerpts allowed), and clearly state any unsupported or contradictory parts if applicable. # Output Format (strictly required) Output **only one JSON object**, with the following three fields: * `"accuracy_score"`: `"0"` or `"1"` or `"2"` * `"is_included_in_golden_memories"`: `"true"` or `"false"` * `"reason"`: `"brief explanation in Chinese"` Do **not** include any other text, explanation, or fields. Do **not** include the candidate memory text inside the JSON. Please output **only** the following JSON (in a code block): ```json {{ "reason": "Brief explanation in Chinese" "accuracy_score": "2 | 1 | 0", "is_included_in_golden_memories": "true | false", }} ``` EVALUATION_PROMPT_FOR_UPDATE_MEMORY: | Your task is to **evaluate the update accuracy** of an AI memory system. Based on the information provided below, determine whether the system-generated **“Generated Memories”** correctly **includes** the **Target Memory for Update**. # Background Information The following information is provided for evaluation: 1. **Generated Memories:** This is the list of memory points generated by the system after the current dialogue. {memories} 2. **Target Memory for Update:** This is the correct, updated version of the memory point that should have been produced — the one we focus on in this evaluation. {updated_memory} 3. **Original Memory Content:** This is the original version of the target memory before the update. {original_memory} # Evaluation Criteria Please make your judgment **strictly based on the content update of the “Target Memory for Update.”** Use the following categories: ### Correct Update * **Generated Memories** **contains all information points** from the “Target Memory for Update,” accurately and completely reflecting the intended update. * **Key fields** (e.g., date, time, values, proper nouns, etc.) must match exactly. * The **original memory** is effectively replaced or marked as outdated. * Synonymous or slightly rephrased expressions are acceptable. ### Hallucinated Update * **Factual error:** The **Generated Memories** includes a new memory related to the “Target Memory for Update,” but its content contains factual mistakes or contradictions compared to the correct update. ### Omitted Update * **Completely omitted:** The **Generated Memories** contains no new memory related to the “Target Memory for Update.” * **Partially omitted:** A related new memory was generated in **Generated Memories**, but it **misses key information** that should have been included. ### Other Used for update failures that do **not clearly fall** into the above categories of “Hallucination” or “Omission.” # Output Requirements Please return your evaluation strictly in the following JSON format and provide a concise explanation. ```json {{ "reason": "Briefly explain your reasoning here and why it fits this category.", "evaluation_result": "Correct | Hallucination | Omission | Other" }} ``` EVALUATION_PROMPT_FOR_QUESTION: | You are an **evaluation expert for AI memory system question answering**. Based **only** on the provided **“Question”**, **“Reference Answer”**, and **“Key Memory Points”** (the essential facts needed to derive the reference answer), strictly evaluate the **accuracy** of the **“Memory System Response.”** Classify it as one of **“Correct”**, **“Hallucination”**, or **“Omission.”** Do **not** use any external knowledge or subjective inference. Finally, output your judgment **strictly** in the specified JSON format. # Evaluation Criteria ## Answer Type Classification ### 1. Correct * The “Memory System Response” accurately answers the “Question,” and its content is **semantically equivalent** to the “Reference Answer.” * It contains **no contradictions** with the “Key Memory Points” or “Reference Answer.” * It introduces **no unsupported details** beyond the “Key Memory Points” that could alter the conclusion. * Synonyms, paraphrasing, and reasonable summarization are acceptable. ### 2. Hallucination * The “Memory System Response” includes information or facts that **contradict or are inconsistent** with the “Reference Answer” or the “Key Memory Points.” * When the “Reference Answer” is labeled as *unknown/uncertain*, yet the response provides a specific verifiable fact or conclusion. * Extra irrelevant information that does **not change** the conclusion is **not** considered hallucination by itself; however, if it **changes or misleads** the conclusion, or **contradicts** the “Key Memory Points,” it should be judged as a **Hallucination**. ### 3. Omission * The response is **incomplete** compared to the “Reference Answer.” * It explicitly states “don’t know,” “can’t remember,” or “no related memory,” even though relevant information exists in the “Key Memory Points.” * For multi-element questions, **all elements must be correct and present**; omission of **any** element is considered an **Omission**. ## Priority Rules (Conflict Handling) * If the response contains **both missing necessary information** and **fabricated/contradictory information**, classify it as **Hallucination**. * If there is **no fabrication/contradiction** but some necessary information is missing, classify it as **Omission**. * Only when the meaning is **fully equivalent** to the reference answer should it be classified as **Correct**. ## Detailed Guidelines and Tolerance * Equivalent expressions of numbers, times, and units are acceptable, but the **numerical values themselves must not differ**. * For multi-element questions, **all elements must be complete and accurate**; missing any element counts as **Omission**. * If the reference answer is *“unknown / cannot be determined”* and the system provides a definite fact, that is a **Hallucination**. If the system also answers *“unknown”* (without guessing), it may be **Correct**. * The evaluation must rely **only** on the *Reference Answer*, *Key Memory Points*, and *System Response* — no external context, world knowledge, or speculative reasoning is allowed. # Information for Evaluation * **Question:** {question} * **Reference Answer:** {reference_answer} * **Key Memory Points:** {key_memory_points} * **Memory System Response:** {response} # Output Requirements Please provide your evaluation result **strictly** in the JSON format below. Do **not** add any extra explanation or comments outside the JSON block. ```json {{ "reasoning": "Provide a concise and traceable evaluation rationale: first compare the system’s response with the Key Memory Points (which were correctly used, which were missing, and whether there was any fabrication/contradiction), then assess its consistency with the Reference Answer, and finally state the classification basis.", "evaluation_result": "Correct | Hallucination | Omission" }} ``` EVALUATION_PROMPT_FOR_QUESTION2: | You are an **evaluation expert for AI memory system question answering**. Based **only** on the provided **"Question"**, **"Reference Answer"**, and **"Key Memory Points"** (the essential facts needed to derive the reference answer), strictly evaluate the **accuracy** of the **"Memory System Response."** Classify it as one of **"Correct"**, **"Hallucination"**, or **"Omission."** Do **not** use any external knowledge or subjective inference. Finally, output your judgment **strictly** in the specified JSON format. # Evaluation Criteria ## Answer Type Classification ### 1. Correct * The "Memory System Response" accurately answers the "Question," and its content is **semantically equivalent** to the "Reference Answer." * It contains **no contradictions** with the "Key Memory Points" or "Reference Answer." * **Extra details not present in the Key Memory Points are allowed and should not be penalized**, as long as they: - Do not contradict the Key Memory Points or Reference Answer - Do not change or mislead the core conclusion - Are reasonable additional context that the memory system may have retained from the conversation * The memory system may have stored additional information beyond the Key Memory Points. Such extra information should be treated as **supplementary context** rather than hallucination, provided it does not conflict with the core answer. * Synonyms, paraphrasing, and reasonable summarization are acceptable. ### 2. Hallucination * The "Memory System Response" includes information or facts that **contradict or are inconsistent** with the "Reference Answer" or the "Key Memory Points." * The response provides information that **directly contradicts** known facts from the Key Memory Points. * When the "Reference Answer" is labeled as *unknown/uncertain*, yet the response provides a specific verifiable fact or conclusion. * **Important:** Extra information that is NOT in Key Memory Points is **NOT automatically a hallucination**. Only classify as hallucination if the extra information: - Directly contradicts the Key Memory Points or Reference Answer - Changes or misleads the core conclusion in a way that makes the answer incorrect - Provides a definitive answer when the Reference Answer indicates uncertainty ### 3. Omission * The response is **incomplete** compared to the "Reference Answer." * It explicitly states "don't know," "can't remember," or "no related memory," even though relevant information exists in the "Key Memory Points." * For multi-element questions, **all elements must be correct and present**; omission of **any** element is considered an **Omission**. ## Priority Rules (Conflict Handling) * If the response contains **both missing necessary information** and **fabricated/contradictory information**, classify it as **Hallucination**. * If there is **no fabrication/contradiction** but some necessary information is missing, classify it as **Omission**. * If the core answer is correct and complete, classify as **Correct** even if there are extra details not in Key Memory Points (as long as they don't contradict or mislead). ## Detailed Guidelines and Tolerance * Equivalent expressions of numbers, times, and units are acceptable, but the **numerical values themselves must not differ**. * For multi-element questions, **all elements must be complete and accurate**; missing any element counts as **Omission**. * If the reference answer is *"unknown / cannot be determined"* and the system provides a definite fact, that is a **Hallucination**. If the system also answers *"unknown"* (without guessing), it may be **Correct**. * **Focus on evaluating whether the core answer to the question is correct**, not whether the response is limited to only the Key Memory Points. * Extra contextual information (e.g., additional preferences, related details) should be viewed as enrichment, not as errors, unless they contradict or mislead. # Information for Evaluation * **Question:** {question} * **Reference Answer:** {reference_answer} * **Key Memory Points:** {key_memory_points} * **Memory System Response:** {response} # Output Requirements Please provide your evaluation result **strictly** in the JSON format below. Do **not** add any extra explanation or comments outside the JSON block. ```json {{ "reasoning": "Provide a concise and traceable evaluation rationale: first verify that the system's response correctly includes all required elements from the Reference Answer, then check if any information contradicts the Key Memory Points or Reference Answer. Extra details not in Key Memory Points should be noted but not penalized unless they contradict or mislead. Finally state the classification basis.", "evaluation_result": "Correct | Hallucination | Omission" }} ``` """